Examining the Stability and Predictive Utility of Across- and Within-Domain Intra-Individual Variability in Mild Cognitive Impairment.

OBJECTIVE: Dispersion is a form of intra-individual variability across neuropsychological tests that has been shown to predict cognitive decline. However, few studies have investigated the stability and predictive utility of both across- and within-domain dispersion. The current study aims to fill these gaps in the literature by examining multiple indices of dispersion in a longitudinal clinical sample of individuals diagnosed with mild cognitive impairment (MCI) at baseline. METHOD: Two hundred thirty-eight MCI patients from a cognitive disorders clinic underwent testing at baseline and after approximately 1.5 years. Linear regression was used to examine whether baseline across- and within-domain dispersion predicted cognitive decline in individuals whose diagnostic classification progressed to dementia (i.e., MCI-Decline) and those who retained an MCI diagnosis at follow-up (i.e., MCI-Stable). Cognitive decline was operationalized dichotomously using group status and continuously using standardized regression-based (SRB) z-scores. RESULTS: Dispersion variables at baseline and follow-up were positively correlated in both groups, with the exception of within-domain executive functioning and language dispersion in the MCI-Decline group. None of the dispersion variables predicted diagnostic conversion to MCI. Using SRB z-scores, greater across-domain dispersion predicted greater overall cognitive decline at follow-up, but this was not the case for within-domain variables with the exception of visuospatial skills. CONCLUSIONS: Results suggest that across- and within-domain dispersion are relatively stable across time, and that across-domain dispersion is predictive of subtle cognitive decline in patients with MCI. However, these results also highlight that findings may differ based on the tests included in dispersion calculations.

Memory changes in patients with hippocampal sclerosis submitted to surgery to treat mesial temporal lobe epilepsy.

PURPOSE: This study was performed with the purpose of analysing the relationship between epileptological and surgical variables and post-operative memory performance, following surgery for refractory mesial temporal lobe epilepsy (MTLE) due to hippocampal sclerosis (HS). METHODS: Logical memory (LM) and visual memory (VM) scores for immediate and late follow-up of 201 patients operated for MTLE/HS were reviewed. Scores were standardized with a control group of 54 healthy individuals matched for age and education. The Reliable Change Index (RCI) was calculated to verify individual memory changes for late LM and VM scores. A multiple linear regression analysis was carried out with the RCI, using LM and VM scores as well as the clinical variables. RESULTS: A total of 112 (56%) patients had right HS. The RCI of the right HS group demonstrated that 6 (7%) patients showed improvement while 5 (6%) patients showed decreased scores in late LM; for late VM, 7 (8%) patients presented improvement, and 2 (3%) patients showed poorer scores. RCI of the left HS group showed that 3 (3%) individuals showed improved scores, while scores of 5 (4%) patients worsened for late LM; for late VM, 3 (3%) patients presented higher scores and 6 (5%) showed lower scores. Left HS and advanced age at onset of the first epileptic seizure were predictors of late LM loss (p<.05). CONCLUSION: Left MTLE/HS and seizure onset at advanced ages were predictive factors for the worsening of late LM. We observed poorer baseline LM function in the left HS group and improvement of LM in some patients who had resection of the right MTL. Patients in the right HS group showed a higher percentage of reliable post-operative improvement for both VM and LM scores.

A longitudinal study of within-person trajectories in quality of life in patients receiving early palliative care.

PURPOSE: To reveal the within-person trajectories in quality of life (QOL) in patients receiving early palliative care. Previous studies have mostly focused aggregated trajectories, based on all research participants combined, whereas this study focused on within-person trajectories in QOL and on whether the variability in QOL trajectories across patients is substantial enough to raise doubts about aggregated trajectories. METHODS: Twenty-five older persons in early palliative care completed the McGill Quality of Life Questionnaire multiple times. Reliable change analyses provided estimates of the occurrence of statistically significant within-person change. RESULTS: There was reliable, within-person variation in QOL scores across time, more so for physical than for psychological aspects of well-being. Changes in QOL scores occurred for most patients but the trajectories were not linear and there was no common trend in the nonlinear patterns. CONCLUSIONS: Reliable change across time can be identified in persons receiving early palliative care. However, the trajectories are notably variable and patient-specific. The high degrees of within-person variability across time suggests the importance of repeated QOL assessments and of dynamic tailoring of clinical treatments.

This study describes the quality of life (QOL) trajectories of 25 persons in early palliative care for life-limiting chronic conditions. Previous studies have mostly focused on aggregated trajectories for all research participants combined. In contrast, this study focused on within-person trajectories in QOL and on whether the variability in QOL trajectories across patients is substantial enough to question the meaningfulness of aggregated trajectories. There was both between-patient and within-person variation in QOL scores across time and no common trend. For some patients, the change was for improving scores across time. For other patients the change was for worsening scores across time. And for still other patients, the changes were reliable but not linear. The trajectories are thus quite variable and patient-specific. The variability in QOL trajectories is substantial enough to raise concerns about aggregated trajectories. The high degrees of within-person variability across time suggests the importance of repeated QOL assessments and of dynamic tailoring of clinical treatments.

Meaningful cognitive change for the Minimal Assessment of Cognitive Function in Multiple Sclerosis.

BACKGROUND: There is limited information on interpretation of cognitive changes over time in multiple sclerosis (MS). OBJECTIVE: This study aimed to provide normative data for the assessment of statistically meaningful change in all tests of the Minimal Assessment of Cognitive Function in MS (MACFIMS). METHODS: We applied the reliable change methodology to a healthy Italian cohort, assessed with two alternate versions of the MACFIMS 1 year apart. We calculated confidence intervals of retest score variance using the reliable change index (RCI). Moreover, multivariable linear regression models adjusted for age, sex, education, and baseline score were built to calculate the regression-based change index (RB-CI). RESULTS: Overall, 200 healthy individuals were enrolled. Thresholds for interpreting change in each test were calculated. In the multivariable models, baseline score was associated with retest score in all tests (B from 0.439 to 0.760; p < 0.001). RB-CI can be calculated with data of the multivariable models. CONCLUSION: We provide normative data for reliable cognitive change evaluation for all the tests of the MACFIMS, which includes the Symbol Digit Modalities Test and Brief International Cognitive Assessment in MS, two widely used tools for screening and monitoring cognition in MS. Our findings can significantly improve the interpretation of cognitive changes in MS.

Changes in overlap of subjective and objective cognition over time in epilepsy surgery candidates.

PURPOSE: Subjective and objective cognition often show weak overlap in persons with epilepsy (PWE). Over- as well as underestimation may occur. In particular after epilepsy surgery, objective memory decline is often not subjectively reported. Additionally, studies on how concordance of subjective and objective cognition changes over time are missing. Therefore, we study the extent of concordance in operated and non-operated PWE over time. METHODS: Candidates for resective epilepsy surgery were assessed between 03/18 and 12/20 (T1) with self-report questionnaires and underwent a neuropsychological examination. For 21 operated as well as 27 non-operated PWE follow-up data was obtained one to three years later (T2). Concordance of attention and memory were compared between groups and time points. Moreover, reliable change was calculated and compared between groups. RESULTS: Of the total sample, 42 % rated their attention performance realistically and 25 % showed memory concordance. Differences in patterns of over- and underestimation between groups and over time occurred for attention, but not for memory. Overestimation of memory was more frequent in operated than non-operated PWE, especially at T2 (67% vs. 11 %). In the operated group, we mainly observed reliable improvement in subjective cognition and decline in objective memory, whereas non-operated PWE showed more frequently decline of simple attention. Reliable subjective and objective change did not co-occur. CONCLUSION: Concordance of subjective and objective cognition is low. Over- as well as underestimation may persist over time. Domain-specific perspectives are necessary. Tendencies of operated PWE to develop overestimation of memory after surgery should be considered in neuropsychological interventions.

Longitudinal Feasibility of the Montreal Cognitive Assessment (MoCA) in Non-Demented ALS Patients.

INTRODUCTION: The present study aimed at testing the longitudinal feasibility of the Montreal Cognitive Assessment (MoCA) in an Italian cohort of non-demented amyotrophic lateral sclerosis (ALS) patients. METHODS: N = 39 non-demented ALS patients were followed-up at a 5-to-10-month interval (M = 6.8; SD = 1.4) with the MoCA and the Edinburgh Cognitive and Behavioral ALS Screen (ECAS). Practice effects, test-retest reliability, and predictive validity (against follow-up ECAS scores) were assessed. Reliable change indices (RCIs) were derived via a regression-based approach by accounting for retest interval and baseline confounders (i.e., demographics, disease duration, and severity and progression rate). RESULTS: At retest, 100% and 69.2% of patients completed the ECAS and the MoCA, respectively. Patients who could not complete the MoCA showed a slightly more severe and fast-progressing disease. The MoCA was not subject to practice effects (t[32] = -0.80; p = 0.429) and was reliable at retest (intra-class correlation = 0.82). Moreover, baseline MoCA scores predicted the ECAS at retest. RCIs were successfully derived - with baseline MoCA scores being the only significant predictor of retest performances (ps < 0.001). CONCLUSIONS: As long as motor disabilities do not undermine its applicability, the MoCA appears to be longitudinally feasible at a 5-to-10-month interval in non-demented ALS patients. However, ALS-specific screeners - such as the ECAS - should be preferred whenever possible.

The Montreal Cognitive Assessment: Norms and Reliable Change Indices for Standard and MoCA-22 Administrations.

OBJECTIVE: The Montreal Cognitive Assessment (MoCA) is among the most frequently administered cognitive screening tests, yet demographically diverse normative data are needed for repeated administrations. METHOD: Data were obtained from 18,410 participants using the National Alzheimer's Coordinating Center Uniform Data Set. We developed regression-based norms using Tobit regression to account for ceiling effects, explored test-retest reliability of total scores and by domain stratified by age and diagnosis with Cronbach's alpha, and reported the cumulative change frequencies for individuals with serial MoCA administrations to gage expected change. RESULTS: Strong ceiling effects and negative skew were observed at the total score, domain, and item levels for the cognitively normal group, and performances became more normally distributed as the degree of cognitive impairment increased. In regression models, years of education was associated with higher MoCA scores, whereas older age, male sex, Black and American Indian or Alaska Native race, and Hispanic ethnicity were associated with lower predicted scores. Temporal stability was adequate and good at the total score level for the cognitively normal and cognitive disorders groups, respectively, but fell short of reliability standards at the domain level. CONCLUSIONS: MoCA total scores are adequately reproducible among those with cognitive diagnoses, but domain scores are unstable. Robust regression-based norms should be used to adjust for demographic performance differences, and the limited reliability, along with the ceiling effects and negative skew, should be considered when interpreting MoCA scores.

Utilizing a learning health system to capture real-world patient data: Application of the reliable change index to evaluate and improve the outcome of a pain rehabilitation program.

BACKGROUND AND OBJECTIVES: The learning healthcare system (LHS) has been developed to integrate patients' clinical data into clinical decisions and improve treatment outcomes. Having little guidance on this integration process, we aim to explain (a) an applicable analytic tool for clinicians to evaluate the clinical outcomes at a group and an individual level and (b) our quality improvement (QI) project, analyzing the outcomes of a new outpatient pain rehabilitation program ("Back-in-Action": BIA) and applying the analysis results to modify our clinical practice. METHODS: Through our LHS (CHOIR; https://choir.stanford.edu), we administered the Pain Catastrophizing Scale (PCS), Chronic Pain Acceptance Questionnaire (CPAQ), and Patient-Reported Outcomes Measures (PROMIS)® before and after BIA. After searching for appropriate analytic tools, we decided to use the Reliable Change Index (RCI) to determine if an observed change in the direction of better (improvement) or worse (deterioration) would be beyond or within the measurement error (no change). RESULTS: Our RCI calculations revealed that at least a 9-point decrease in the PCS scores and 10-point increase in the CPAQ scores would indicate reliable improvement. RCIs for the PROMIS measures ranged from 5 to 8 T-score points (i.e., 0.5-0.8 SD). When evaluating change scores of the PCS, CPAQ, and PROMIS measures, we found that 94% of patients showed improvement in at least one domain after BIA and 6% showed no reliable improvement. CONCLUSIONS: Our QI project revealed RCI as a useful tool to evaluate treatment outcomes at a group and an individual level, and RCI could be incorporated into the LHS to generate a progress report automatically for clinicians. We further explained how clinicians could use RCI results to modify a clinical practice, to improve the outcomes of a pain program, and to develop individualized care plans. Lastly, we suggested future research areas to improve the LHS application in pain practice.

Adherence to the MIND diet and the odds of mild cognitive impairment in generally healthy older adults: The 3-year DO-HEALTH study.

BACKGROUND: The Mediterranean-DASH Intervention for Neurodegenerative Delay (MIND) diet may slow cognitive decline in older adults. A potential mechanism could be possible anti-inflammatory properties of the MIND-diet. OBJECTIVE: To examine whether adherence to the MIND diet at baseline is associated with the odds of mild cognitive impairment (MCI) and changes in biomarkers of inflammation (High-sensitivity C-reactive Protein(hsCRP), interleukin-6(IL-6)) over three years in adults ≥70 years. METHODS: Adherence to the MIND diet was assessed by food frequency questionnaire (FFQ) at baseline and after three years. Presence of MCI based on the Montreal Cognitive Assessment (MoCA) was defined as <26 (MCI26), or <24 (MCI24). We performed a minimally adjusted model controlling for sex, prior fall, linear spline at age 85, time, treatment and study site. The fully adjusted model also adjusted for education, BMI, physical activity, depression score, daily energy intake, and comorbidity score. To assess the change in inflammatory markers from baseline, we used linear-mixed-effect models adjusted for the same variables plus the respective baseline concentrations. Sensitivity analyses accounting for practice effects of repeated cognitive tests using the reliable change index for both MoCA cut-offs were done. RESULTS: We included 2028 of 2157 DO-HEALTH participants (60.5% women; mean age 74.88 years) with complete data. Adherence to the MIND diet at baseline was not associated with cognitive decline over three years, neither at MoCA < 26 (OR (95%CI) = 0.99 (0.94-1.04)) nor at MoCA < 24 (OR (95%CI) = 1.03 (0.96-1.1)). Applying the reliable change index to the two cut-offs confirmed the findings. Further, the MIND diet adherence was not associated with the change in MoCA score from baseline in DO-HEALTH. For inflammatory biomarkers MIND-diet baseline adherence was not associated with changes in hsCRP or IL-6. CONCLUSION: Adherence to the MIND-diet was neither associated with the odds of MCI, nor with hsCRP or IL-6 at baseline. Moreover, change in MIND-diet over three years was not associated with changes in hsCRP or IL-6.

Test-retest reliability and reliable change index of the Philips IntelliSpace Cognition digital test battery.

OBJECTIVE: This article provides the test-retest reliability and Reliable Change Indices (RCIs) of the Philips IntelliSpace Cognition (ISC) platform, which contains digitized versions of well-established neuropsychological tests. METHOD: 147 participants (ages 19 to 88) completed a digital cognitive test battery on the ISC platform or paper-pencil versions of the same test battery during two separate visits. Intraclass correlation coefficients (ICC) were calculated separately for the ISC and analog test versions to compare reliabilities between administration modalities. RCIs were calculated for the digital tests using the practice-adjusted RCI and standardized regression-based (SRB) method. RESULTS: Test-retest reliabilities for the ISC tests ranged from moderate to excellent and were comparable to the test-retest reliabilities for the paper-pencil tests. Baseline test performance, retest interval, age, and education predicted test performance at visit 2 with baseline test performance being the strongest predictor for all outcome measures. For most outcome measures, both methods for the calculation of RCIs show agreement on whether or not a reliable change was observed. CONCLUSIONS: RCIs for the digital tests enable clinicians to determine whether a measured change between assessments is due to real improvement or decline. Together, this contributes to the growing evidence for the clinical utility of the ISC platform.

Revisiting reliable change with Iverson (2001).

Objective: Iverson (2001) expanded on reliable change methodology by accounting for the variability in scores at Time 2 in the calculation of change scores. However, due to limitations in available data, an incomplete picture of variables affecting change was presented. The current paper sought to address some of these limitations and clarify the methodology for assessing reliable change. Method: Using one-year test-retest data on the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) in older adults who were cognitively intact or had mild Alzheimer's disease (AD), change scores were calculated, and various models, consistent with or divergent from Iverson, were presented. Results: Across the RBANS, individuals with intact cognition tended to show less variable scores, especially at Time 2, which resulted in larger change scores than those with AD. When applied to an independent sample, different patterns of change were observed, with: (1) models that used intact data showed more cognitive change than those using AD data; (2) the model that corrected for practice effects and used intact data showed the most decline; and (3) the model that corrected for practice effects and used AD data showed the most improvement. The models that showed the strongest association in classifying independent cases as decline/stable/improve were those that used intact data and were discordant on the use of practice effects. Conclusions: Overall, findings highlight the complexity of calculating reliable change, and they lend additional caution to Iverson's original limitations. However, the use of data from individuals classified as cognitively intact and a correction for practice effects seems warranted.

Relationship of Hoarding and Depression Symptoms in Older Adults.

Hoarding disorder (HD) is a debilitating neuropsychiatric condition that affects 2%-6% of the population and increases in incidence with age. Major depressive disorder (MDD) co-occurs with HD in approximately 50% of cases and leads to increased functional impairment and disability. However, only one study to date has examined the rate and trajectory of hoarding symptoms in older individuals with a lifetime history of MDD, including those with current active depression (late-life depression; LLD). We therefore sought to characterize this potentially distinct phenotype. We determined the incidence of HD in two separate cohorts of participants with LLD (n = 73) or lifetime history of MDD (n = 580) and examined the reliability and stability of hoarding symptoms using the Saving Inventory-Revised (SI-R) and Hoarding Rating Scale-Self Report (HRS), as well as the co-variance of hoarding and depression scores over time. HD was present in 12% to 33% of participants with MDD, with higher rates found in those with active depressive symptoms. Hoarding severity was stable across timepoints in both samples (all correlations >0.75), and fewer than 30% of participants in each sample experienced significant changes in severity between any two timepoints. Change in depression symptoms over time did not co-vary with change in hoarding symptoms. These findings indicate that hoarding is a more common comorbidity in LLD than previously suggested, and should be considered in screening and management of LLD. Future studies should further characterize the interaction of these conditions and their impact on outcomes, particularly functional impairment in this vulnerable population.

Exploring the Number of Web-Based Behavioral Health Coaching Sessions Associated With Symptom Improvement in Youth: Observational Retrospective Analysis.

BACKGROUND: Rates of anxiety and depression have been increasing among children and adolescents for the past decade; however, many young people do not receive adequate mental health care. Digital mental health interventions (DMHIs) that include web-based behavioral health coaching are widely accessible and can confer significant improvements in youth anxiety and depressive symptoms. However, more research is necessary to determine the number of web-based coaching sessions that confer clinically significant improvements in anxiety and depressive symptoms in youth. OBJECTIVE: This study uses data from a pediatric DMHI to explore the number of web-based coaching sessions required to confer symptom improvements among children and adolescents with moderate or moderately severe symptoms of anxiety and depression. METHODS: We used retrospective data from a pediatric DMHI that offered web-based behavioral health coaching in tandem with self-guided access to asynchronous chat with practitioners, digital mental health resources, and web-based mental health symptom assessments. Children and adolescents who engaged in 3 or more sessions of exclusive behavioral health coaching for moderate to moderately severe symptoms of anxiety (n=66) and depression (n=59) were included in the analyses. Analyses explored whether participants showed reliable change (a decrease in symptom scores that exceeds a clinically established threshold) and stable reliable change (at least 2 successive assessments of reliable change). Kaplan-Meier survival analyses were performed to determine the median number of coaching sessions when the first reliable change and stable reliable change occurred for anxiety and depressive symptoms. RESULTS: Reliable change in anxiety symptoms was observed after a median of 2 (95% CI 2-3) sessions, and stable reliable change in anxiety symptoms was observed after a median of 6 (95% CI 5-8) sessions. A reliable change in depressive symptoms was observed after a median of 2 (95% CI 1-3) sessions, and a stable reliable change in depressive symptoms was observed after a median of 6 (95% CI 5-7) sessions. Children improved 1-2 sessions earlier than adolescents. CONCLUSIONS: Findings from this study will inform caregivers and youth seeking mental health care by characterizing the typical time frame in which current participants show improvements in symptoms. Moreover, by suggesting that meaningful symptom improvement can occur within a relatively short time frame, these results bolster the growing body of research that indicates web-based behavioral health coaching is an effective form of mental health care for young people.

Health Economic Evaluation of an Online-Based Motivational Program to Reduce Problematic Media Use and Promote Treatment Motivation for Internet Use Disorder-Results of the OMPRIS Study.

Internet Use Disorders (IUD) have a relevant effect on national economies. In the randomized, controlled, multicenter, prospective, and single-blinded OMPRIS study (pre-registration number DRKS00019925; Innovation Fund of the Joint Federal Committee of Germany, grant number 01VSF18043), a four-week online program to reduce media addiction symptoms, was evaluated for cost-effectiveness. The intervention group (IG) was compared to a waiting control group (WCG) from German statutory health insurance (SHI) and a societal perspective. Resource use, namely indirect and direct (non) medical costs, was assessed by a standardized questionnaire at baseline and after the intervention. Additionally, intervention costs were calculated. Determining the Reliable Change Index (RCI) based on the primary outcome, assessed by the "Scale for the Assessment of Internet and Computer Game Addiction" (AICA-S), individuals with and without reliable change (RC) were distinguished. The incremental cost-effectiveness ratio was calculated using the difference-in-difference approach. There were 169 (IG n = 81, WCG n = 88) persons included in the analysis. The mean age was 31.9 (SD 12.1) years. A total of 75.1% were male, and 1.8% diverse. A total of 65% (IG) and 27% (WCG) had an RC. The cost per person with RC was about EUR 860 (SHI) and EUR 1110 (society). The intervention leads to an improvement of media addiction symptoms at moderate additional costs.

Detecting Change in Needs-Supplies Fit Through Reliable Change Methodology.

Studying change is a critical part of psychology and science in general. Studies often treat fit as static and use between-person designs to assess change. Accordingly, potentially insightful within-person information is frequently overlooked. The current study aimed to establish the utility of reliable change methodology within the domain of organizational psychology, using needs-supplies fit as a guiding framework. When employee needs can be tracked with a fair degree of clarity, organizations can devise better strategies to routinely address discrepancies between desired employee needs and organizational offerings. This longitudinal study used secondary data from 258 secondary school teachers. The Needs-Supplies Fit Scale was administered. The study's hypothesis was assessed through reliable change methodology. When considering all the participants that experienced at least some change across time intervals (n = 148), 23.65% (n = 35) of this change was meaningful. This declined to 17.33% meaningful change when factoring in the entire sample, including those who did not experience any change. When organizations are aware of the ever-evolving needs of employees, quicker action can be taken to avoid impending person-environment misfit. This study contributes to existing within-person studies that showcased the malleability of needs-supplies fit and emphasize the value of placing a more prominent focus on the individual.

Recent Advances in Neuropsychological Test Interpretation for Clinical Practice.

Much attention in the field of clinical neuropsychology has focused on adapting to the modern healthcare environment by advancing telehealth and promoting technological innovation in assessment. Perhaps as important (but less discussed) are advances in the development and interpretation of normative neuropsychological test data. These techniques can yield improvement in diagnostic decision-making and treatment planning with little additional cost. Brooks and colleagues (Can Psychol 50: 196-209, 2009) eloquently summarized best practices in normative data creation and interpretation, providing a practical overview of norm development, measurement error, the base rates of low scores, and methods for assessing change. Since the publication of this seminal work, there have been several important advances in research on development and interpretation of normative neuropsychological test data, which may be less familiar to the practicing clinician. Specifically, we provide a review of the literature on regression-based normed scores, item response theory, multivariate base rates, summary/factor scores, cognitive intraindividual variability, and measuring change over time. For each topic, we include (1) an overview of the method, (2) a rapid review of the recent literature, (3) a relevant case example, and (4) a discussion of limitations and controversies. Our goal was to provide a primer for use of normative neuropsychological test data in neuropsychological practice.

Identifying thresholds for clinically meaningful change among clients of drug and alcohol services using the Australian Treatment Outcomes Profile.

AIMS: The Austraian Treatment Outcomes Profile (ATOP) is a brief clinical outcomes tool used widely in the Australian alcohol and other drugs treatment sector to monitor clients' substance use, health, wellbeing and clinical risk factors. It has demonstrated reliability and validity, and has recommended clinical cut-offs for assessing single-occasion client-rated health scores. This study determined clinically meaningful change thresholds for ATOP substance use and health and wellbeing variables for use by clinicians in monitoring client progress, and for quality improvement and service evaluation. DESIGN, SETTING AND PARTICIPANTS: A framework for assessing clinically meaningful changes scores was developed by (1) calculating statistically reliable change thresholds using data-driven techniques with a reference sample of clinical ATOP data and (2) conducting a multi-disciplinary subject matter expert group to review the utility and validity of data-derived clinically meaningful change. The study was conducted within Outpatient Alcohol and Other Drug treatment services in New South Wales, Australia. The reference sample comprised 6100 ATOPs from clients at entry to public outpatient Alcohol and Other Drug treatment services; the subject matter expert group comprised 29 key stakeholders from the specialist alcohol and other drug treatment sector. MEASUREMENTS AND FINDINGS: We used the Reliable Change Index method to calculate clinically meaningful change thresholds for ATOP variables. For substance use variables, a change of 30% in days of use in the last 28 (minimum 4 days) was the threshold for clinically meaningful change for substance use; for health and wellbeing variables, a change of 2 or more points in psychological health, physical health or quality of life scores (measured on 0-10 scales) was the minimum clinically meaningful change. CONCLUSIONS: Clinically meaningful change thresholds have been proposed for Australian Treatment Outcomes Profile substance use and health and wellbeing items, based on statistical reliability and subject matter expert assessment. These will be used in the development of an outcomes metric for assessing change and assigning meaning in aggregated data for evaluation of services.

TEST-RETEST RELIABILITY AND RELIABLE CHANGE INDEX OF MOBILE APPLICATION NEUROCOGNITIVE TEST AMONG MIDDLE AND HIGH SCHOOL ATHLETES.

CONTEXT: A mobile application neurocognitive assessment has been used in place of equipment intensive computerized neurocognitive assessment protocol. A previous study reported high to very high test-retest reliability of neurocognitive assessment using the mobile application in healthy adults, but no studies have examined test-retest reliability, reliable change indices (RCIs), and sex effect in middle school and high school populations when conducted 1 year apart. OBJECTIVE: The purpose of this study was to examine the test-retest reliability and RCIs of baseline data collected at 2-time points approximately 1 year apart using a mobile application neurocognitive rest in middle school and high school athletes. The secondary purpose of the study was to investigate the sex difference in neurocognitive measures. DESIGN: Cross-sectional study. SETTING: Institutional. PATIENTS OR OTHER PARTICIPANTS: 172 middle school and high school healthy student-athletes (mean age=13.78±1.59 years old). MAIN OUTCOME MEASURE(S): Mobile application neurocognitive rest scores (reaction time, impulse control, inspection, and memory). RESULTS: The result from the study demonstrated that neurocognitive measures had low test-retest reliability across a 1-year time period in middle and high school settings. Upon retesting, reaction time and inspection time improved significantly in both middle and high school athletes, and impulse control showed significant improvement in middle school athletes. More athletes in middle school showed more RCI improvements compared to high school athletes. While both males and females demonstrated improvements in neurocognitive measures throughout adolescence, males outperformed females on reaction time and impulse control. CONCLUSIONS: Findings from the study indicate unacceptably low test-retest reliability of a mobile application neurocognitive test most likely due to cognitive development occurring throughout adolescence. Additionally, significant RCIs were noted. These naturally occurring improvements due to cognitive development could mask the post-concussion deficits. The findings warrant consideration of age and sex on the neurocognitive performance of middle and high school athletes.

Comparison of false positive and false negative rates of two indices of individual reliable change: Jacobson-Truax and Hageman-Arrindell methods.

Background: Quantification of change is crucial for correctly estimating the effect of a treatment and for distinguishing random or non-systematic changes from substantive changes. The objective of the present study was to learn about the performance of two distribution-based methods [the Jacobson-Truax Reliable Change Index (RCI) and the Hageman-Arrindell (HA) approach] that were designed for evaluating individual reliable change. Methods: A pre-post design was simulated with the purpose to evaluate the false positive and false negative rates of RCI and HA methods. In this design, a first measurement is obtained before treatment and a second measurement is obtained after treatment, in the same group of subjects. Results: In relation to the rate of false positives, only the HA statistic provided acceptable results. Regarding the rate of false negatives, both statistics offered similar results, and both could claim to offer acceptable rates when Ferguson's stringent criteria were used to define effect sizes as opposed to when the conventional criteria advanced by Cohen were employed. Conclusion: Since the HA statistic appeared to be a better option than the RCI statistic, we have developed and presented an Excel macro so that the greater complexity of calculating HA would not represent an obstacle for the non-expert user.