Recommendations on the Selection, Development, and Modification of Performance Outcome Assessments: A Good Practices Report of an ISPOR Task Force.

In evaluating the clinical benefit of new therapeutic interventions, it is critical that the treatment outcomes assessed reflect aspects of health that are clinically important and meaningful to patients. Performance outcome (PerfO) assessments are measurements based on standardized tasks actively undertaken by a patient that reflect physical, cognitive, sensory, and other functional skills that bring meaning to people's lives. PerfO assessments can have substantial value as drug development tools when the concepts of interest being measured best suit task performance and in cases where patients may be limited in their capacity for self-report. In their development, selection, and modification, including the evaluation and documentation of validity, reliability, usability, and interpretability, the good practice recommendations established for other clinical outcome assessment types should continue to be followed, with concept elicitation as a critical foundation. In addition, the importance of standardization, and the need to ensure feasibility and safety, as well as their utility in patient groups, such as pediatric populations, or those with cognitive and psychiatric challenges, may enhance the need for structured pilot evaluations, additional cognitive interviewing, and evaluation of quantitative data, such as that which would support concept confirmation or provide ecological evidence and other forms of construct evidence within a unitary approach to validity. The opportunity for PerfO assessments to inform key areas of clinical benefit is substantial and establishing good practices in their selection or development, validation, and implementation, as well as how they reflect meaningful aspects of health is critical to ensuring high standards and in furthering patient-focused drug development.

Recommendations to address key recruitment challenges of Alzheimer's disease clinical trials.

Clinical trials for Alzheimer's disease (AD) are slower to enroll study participants, take longer to complete, and are more expensive than trials in most other therapeutic areas. The recruitment and retention of a large number of qualified, diverse volunteers to participate in clinical research studies remain among the key barriers to the successful completion of AD clinical trials. An advisory panel of experts from academia, patient-advocacy organizations, philanthropy, non-profit, government, and industry convened in 2020 to assess the critical challenges facing recruitment in Alzheimer's clinical trials and develop a set of recommendations to overcome them. This paper briefly reviews existing challenges in AD clinical research and discusses the feasibility and implications of the panel's recommendations for actionable and inclusive solutions to accelerate the development of novel therapies for AD.

Psychometric evaluation of the Work Readiness Questionnaire in schizophrenia.

OBJECTIVE/INTRODUCTION: Unemployment can negatively impact quality of life among patients with schizophrenia. Employment status depends on ability, opportunity, education, and cultural influences. A clinician-rated scale of work readiness, independent of current work status, can be a valuable assessment tool. A series of studies were conducted to create and validate a Work Readiness Questionnaire (WoRQ) for clinicians to assess patient ability to engage in socially useful activity, independent of work availability. METHODS: Content validity, test-retest and inter-rater reliability, and construct validity were evaluated in three separate studies. RESULTS: Content validity was supported. Cronbach's α was 0.91, in the excellent range. Clinicians endorsed WoRQ concepts, including treatment adherence, physical appearance, social competence, and symptom control. The final readiness decision showed good test-retest reliability and moderate inter-rater reliability. Work readiness was associated with higher function and lower levels of negative symptoms. Low positive and high negative predictive values confirmed the concept validity. DISCUSSION: The WoRQ has suitable psychometric properties for use in a clinical trial for patients with a broad range of symptom severity. The scale may be applicable to assess therapeutic interventions. It is not intended to assess eligibility for supported work interventions. CONCLUSIONS: The WoRQ is suitable for use in schizophrenia clinical trials to assess patient work functional potential.

Methods for Neuroscience Drug Development: Guidance on Standardization of the Process for Defining Clinical Outcome Strategies in Clinical Trials.

Neurosciences clinical trials continue to have notoriously high failure rates. Appropriate outcomes selection in early clinical trials is key to maximizing the likelihood of identifying new treatments in psychiatry and neurology. The field lacks good standards for designing outcome strategies, therefore The Outcomes Research Group was formed to develop and promote good practices in outcome selection. This article describes the first published guidance on the standardization of the process for clinical outcomes in neuroscience. A minimal step process is defined starting as early as possible, covering key activities for evidence generation in support of content validity, patient-centricity, validity requirements and considerations for regulatory acceptance. Feedback from expert members is provided, regarding the risks of shortening the process and examples supporting the recommended process are summarized. This methodology is now available to researchers in industry, academia or clinics aiming to implement consensus-based standard practices for clinical outcome selection, contributing to maximizing the efficiency of clinical research.

Sensitivity of Individual and Composite Test Scores from the Cogstate Brief Battery to Mild Cognitive Impairment and Dementia Due to Alzheimer's Disease.

BACKGROUND: The Cogstate Brief Battery (CBB) is a computerized cognitive test battery used commonly to identify cognitive deficits related to Alzheimer's disease (AD). However, AD and normative samples used to understand the sensitivity of the CBB to AD in the clinic have been limited, as have the outcome measures studied. OBJECTIVE: This study investigated the sensitivity of CBB outcomes, including potential composite scores, to cognitive impairment in mild cognitive impairment (MCI) and dementia due to AD, in carefully selected samples. METHODS: Samples consisted of 4,871 cognitively unimpaired adults and 184 adults who met clinical criteria for MCI (Clinical Dementia Rating (CDR) = 0.5) or dementia (CDR > 0.5) due to AD and CBB naive. Speed and accuracy measures from each test were examined, and theoretically- and statistically-derived composites were created. Sensitivity and specificity of classification of cognitive impairment were compared between outcomes. RESULTS: Individual CBB measures of learning and working memory showed high discriminability for AD-related cognitive impairment for CDR 0.5 (AUCs ∼ 0.79-0.88), and CDR > 0.5 (AUCs ∼ 0.89-0.96) groups. Discrimination ability for theoretically derived CBB composite measures was high, particularly for the Learning and Working Memory (LWM) composite (CDR 0.5 AUC = 0.90, CDR > 0.5 AUC = 0.97). As expected, statistically optimized linear composite measures showed strong discrimination abilities albeit similar to the LWM composite. CONCLUSIONS: In older adults, the CBB is effective for discriminating cognitive impairment due to MCI or AD-dementia from unimpaired cognition with the LWM composite providing the strongest sensitivity.

Clinically Relevant Changes for Cognitive Outcomes in Preclinical and Prodromal Cognitive Stages: Implications for Clinical Alzheimer Trials.

BACKGROUND AND OBJECTIVES: Identifying a clinically meaningful change in cognitive test score is essential when using cognition as an outcome in clinical trials. This is especially relevant because clinical trials increasingly feature novel composites of cognitive tests. Our primary objective was to establish minimal clinically important differences (MCIDs) for commonly used cognitive tests, using anchor-based and distribution-based methods, and our secondary objective was to investigate a composite cognitive measure that best predicts a minimal change in the Clinical Dementia Rating-Sum of Boxes (CDR-SB). METHODS: From the Swedish BioFINDER cohort study, we consecutively included cognitively unimpaired (CU) individuals with and without subjective or mild cognitive impairment (MCI). We calculated MCIDs associated with a change of ≥0.5 or ≥1.0 on CDR-SB for Mini-Mental State Examination (MMSE), ADAS-Cog delayed recall 10-word list, Stroop, Letter S Fluency, Animal Fluency, Symbol Digit Modalities Test (SDMT) and Trailmaking Test (TMT) A and B, and triangulated MCIDs for clinical use for CU, MCI, and amyloid-positive CU participants. For investigating cognitive measures that best predict a change in CDR-SB of ≥0.5 or ≥1.0 point, we conducted receiver operating characteristic analyses. RESULTS: Our study included 451 cognitively unimpaired individuals, 90 with subjective cognitive decline and 361 without symptoms of cognitive decline (pooled mean follow-up time 32.4 months, SD 26.8, range 12-96 months), and 292 people with MCI (pooled mean follow-up time 19.2 months, SD 19.0, range 12-72 months). We identified potential triangulated MCIDs (cognitively unimpaired; MCI) on a range of cognitive test outcomes: MMSE -1.5, -1.7; ADAS delayed recall 1.4, 1.1; Stroop 5.5, 9.3; Animal Fluency: -2.8, -2.9; Letter S Fluency -2.9, -1.8; SDMT: -3.5, -3.8; TMT A 11.7, 13.0; and TMT B 24.4, 20.1. For amyloid-positive CU, we found the best predicting composite cognitive measure included gender and changes in ADAS delayed recall, MMSE, SDMT, and TMT B. This produced an AUC of 0.87 (95% CI 0.79-0.94, sensitivity 75%, specificity 88%). DISCUSSION: Our MCIDs may be applied in clinical practice or clinical trials for identifying whether a clinically relevant change has occurred. The composite measure can be useful as a clinically relevant cognitive test outcome in preclinical AD trials.

Cognitive performance in relapsing remitting multiple sclerosis: a longitudinal study in daily practice using a brief computerized cognitive battery.

BACKGROUND: There is need for a cognitive test battery that can be easily used in clinical practice to detect or monitor cognitive performance in patients with multiple sclerosis (MS). In order to conduct, in this patient group, a preliminary investigation of the validity and utility of a brief computerized battery, the Cognitive Drug Research (CDR) battery, we longitudinally assessed cognition in patients with relapsing remitting (RR) MS. METHODS: Forty-three mildly disabled, clinically active RRMS patients were repeatedly assessed with the Digit Symbol Substitution Test (DSST), Paced Auditory Serial Addition Test (PASAT) and five composite scores derived from the CDR computerized cognitive test system (CDR System): Power of Attention, Continuity of Attention, Quality of Working Memory, Quality of Episodic Memory and Speed of Memory. The Multiple Sclerosis Functional Composite (MSFC) and Expanded Disability Status Scale (EDSS) measured disability. RESULTS: The composite scores from the CDR battery generally showed excellent test-retest reliability over the repeated assessments, though was low on occasions for the Quality of Working Memory and Quality of Episodic Memory measures. The CDR measures tended to be highly correlated with other measures of cognition (DSST and PASAT) and were also strongly related to disability (EDSS and MSFC). Baseline scores indicated large impairments to visual information processing speed and attention (DSST, Cohen's d 1.1; Power of Attention d 1.4 [reaction time on tasks of focussed and sustained attention]), and a moderate impairment both to sustained attention (Continuity of Attention d 0.6) and complex information processing speed (Speed of memory d 0.7 [reaction time on tasks of working and episodic Memory]), when compared to normative data derived from healthy volunteers enrolled in a series of separate, prior clinical trials. Working memory (Quality of Working Memory) and episodic memory (Quality of Episodic Memory) were unimpaired. CONCLUSIONS: Preliminary validation of the CDR System indicated that for most, but not all measures psychometric properties were adequate and the measures were related to disability (EDSS and MSFC) and other measures of cognition.

Cognitive Go/No-Go decision-making criteria in Alzheimer's disease drug development.

Go/No-Go decision making in early phase clinical trials is challenging for drug developers working in Alzheimer's disease. Recent negative trial results have been attributed to a lack of efficacy and important safety concerns. Furthermore, demonstrated target engagement has rarely translated into demonstrable clinical efficacy. Cognitive data might provide valuable insights at various points during drug development, and a thoughtful and robust set of decision-making criteria, specified a priori, can and should be applied under many circumstances. This review provides insights into how to utilize cognitive data for Go/No-Go decisions, with an emphasis on how these cognitive criteria differ depending on the context (e.g., stage of development, mechanism of action and trial design).

A Paradox in Digital Memory Assessment: Increased Sensitivity With Reduced Difficulty.

The One Card Learning Test (OCL80) from the Cogstate Brief Battery-a digital cognitive test used both in-person and remotely in clinical trials and in healthcare contexts to inform health decisions-has shown high sensitivity to changes in memory in early Alzheimer's disease (AD). However, recent studies suggest that OCL sensitivity to memory impairment in symptomatic AD is not as strong as that for other standardized assessments of memory. This study aimed to improve the sensitivity of the OCL80 to AD-related memory impairment by reducing the test difficultly (i.e., OCL48). Experiment 1 showed performance in healthy adults improved on the OCL48 while the pattern separation operations that constrain performance on the OCL80 were retained. Experiment 2 showed repeated administration of the OCL48 at short retest intervals did not induce ceiling or practice effects. Experiment 3 showed that the sensitivity of the OCL48 to AD-related memory impairment (Glass's Δ = 3.11) was much greater than the sensitivity of the OCL80 (Glass's Δ = 1.94). Experiment 4 used data from a large group of cognitively normal older adults to calibrate performance scores between the OCL80 and OCL48 using equipercentile equating. Together these results showed the OCL48 to be a valid and reliable test of learning with greater sensitivity to memory impairment in AD than the OCL80.

Pilot Evaluation of the Unsupervised, At-Home Cogstate Brief Battery in ADNI-2.

BACKGROUND: There is a need for feasible, scalable assessments to detect cognitive impairment and decline. The Cogstate Brief Battery (CBB) is validated for Alzheimer's disease (AD) and in unsupervised and bring your own device contexts. The CBB has shown usability for self-completion in the home but has not been employed in this way in a multisite clinical trial in AD. OBJECTIVE: The objective of the pilot was to evaluate feasibility of at-home, self-completion of the CBB in the Alzheimer's Disease Neuroimaging Initiative (ADNI) over 24 months. METHODS: The CBB was included as a pilot for cognitively normal (CN) and mild cognitive impairment (MCI) participants in ADNI-2, invited to take the assessment in-clinic, then at at-home over a period of 24 months follow-up. Data were analyzed to explore acceptability/usability, concordance of in-clinic and at-home assessment, and validity. RESULTS: Data were collected for 104 participants (46 CN, 51 MCI, and 7 AD) who consented to provide CBB data. Subsequent analyses were performed for the CN and MCI groups only. Test completion rates were 100%for both the first in-clinic supervised and first at-home unsupervised assessments, with few repeat performances required. However, available follow-up data declined sharply over time. Good concordance was seen between in-clinic and at-home assessments, with non-significant and small effect size differences (Cohen's d between -0.04 and 0.28) and generally moderate correlations (r = 0.42 to 0.73). Known groups validity was also supported (11/16 comparisons with Cohen's d≥0.3). CONCLUSION: These data demonstrate the feasibility of use for the CBB for unsupervised at-home, testing, including MCI groups. Optimal approaches to the application of assessments to support compliance over time remain to be determined.

Building clinically relevant outcomes across the Alzheimer's disease spectrum.

Demonstrating that treatments are clinically meaningful across the Alzheimer's disease (AD) continuum is critical for meeting our goals of accelerating a cure by 2025. While this topic has been a focus of several Alzheimer's Association Research Roundtable (AARR) meetings, there remains no consensus as to what constitutes a "clinically meaningful outcome" in the eyes of patients, clinicians, care partners, policymakers, payers, and regulatory bodies. Furthermore, the field has not come to agreement as to what constitutes a clinically meaningful treatment effect at each stage of disease severity. The AARR meeting on November 19-20, 2019, reviewed current approaches to defining clinical meaningfulness from various perspectives including those of patients and care partners, clinicians, regulators, health economists, and public policymakers. Participants discussed approaches that may confer clinical relevance at each stage of the disease continuum and fostered discussion about what should guide us in the future.

Cognition assessment in paediatric clinical trials.

As in adults and the elderly, a number of paediatric conditions have a recognised cognitive dysfunction. Studies also demonstrate adverse cognitive effects associated with medicinal products used in paediatric populations. Demonstrating efficacy in treating cognitive dysfunction, or safety and tolerability in minimising adverse cognitive effects is as relevant to paediatric clinical trials as it is to adult populations. Indeed, cognitive assessments may be even more vital in paediatric populations, due to the additional possibility of adverse effects on cognitive development. This requirement to evaluate cognition in children and adolescents, and new legislation for paediatric clinical trials, has created an increased demand for suitable assessment methods.

Outcomes Assessment in Clinical Trials of Alzheimer's Disease and its Precursors: Readying for Short-term and Long-term Clinical Trial Needs.

An evolving paradigm shift in the diagnostic conceptualization of Alzheimer's disease is reflected in its recently updated diagnostic criteria from the National Institute on Aging-Alzheimer's Association and the International Working Group. Additionally, it is reflected in the increased focus in this field on conducting prevention trials in addition to improving cognition and function in people with dementia. These developments are making key contributions towards defining new regulatory thinking around Alzheimer's disease treatment earlier in the disease continuum. As a result, the field as a whole is now concentrated on exploring the next-generation of cognitive and functional outcome measures that will support clinical trials focused on treating the slow slide into cognitive and functional impairment. With this backdrop, the International Society for CNS Clinical Trials and Methodology convened semi-annual working group meetings which began in spring of 2012 to address methodological issues in this area. This report presents the most critical issues around primary outcome assessments in Alzheimer's disease clinical trials, and summarizes the presentations, discussions, and recommendations of those meetings, within the context of the evolving landscape of Alzheimer's disease clinical trials.

The effect of desvenlafaxine on cognitive functioning in employed outpatients with major depressive disorder: a substudy of a randomized, double-blind, placebo-controlled trial.

The objective of this substudy was to examine the effect of desvenlafaxine 50 mg/day compared with placebo on cognitive function in employed outpatients with major depressive disorder. A total of 11/55 (20%) study sites in a 12-week, randomized, double-blind, placebo-controlled trial administered cognitive assessments in memory, attention, and executive functioning domains using the cognitive drug research system. Changes from baseline were subjected to analysis of covariance with baseline levels as covariates, using last observation carried forward data. A significant improvement with desvenlafaxine 50 mg/day (n=52) compared with placebo (n=29) was observed on the quality of working memory composite measure (0.081 units (0.005, 0.156); P=0.0365) at last observation carried forward. Improvement from baseline on the speed of working memory composite was significant for desvenlafaxine (-226.6 msec (-316.7, -136.4); P<0.0001) and for placebo (-133.3 msec (-257.2, -9.4); P=0.0354); however, the treatment effect was not significant. No significant differences between groups were observed on composite measures for attention. Treatment of depression with desvenlafaxine 50 mg/day may improve aspects of cognitive functioning, including working memory.Clinical Trial Registry No.: Clinicaltrials.gov identifier: NCT00824291.

Establishing Equivalence of Electronic Clinician-Reported Outcome Measures.

BACKGROUND: Electronic administration of clinician-reported outcomes (eClinROs) has advantages over paper-based methods, but the mode of administration change has the potential to affect the validity of the scale. The literature on migration of patient-reported outcomes (PROs) suggests that there are different levels of modification, which necessitate different approaches to demonstrating mode equivalence. However, little has been written on the migration of ClinROs to electronic administration. METHODS: We propose a method of comparing paper and electronic versions of scales that includes a comparison based on content and a comparison based on format. The determination of whether the eClinRO has undergone minor, moderate, or substantial modification will drive the necessary studies required for validation. RESULTS: The unique characteristics of ClinROs suggest 2 additional types of modifications, including functionality adaptation and adaptation of instructions. CONCLUSIONS: In many respects, the migration of a ClinRO to electronic administration is similar to that of a PRO. This article has explored the ways in which there might be special considerations for ClinROs that have not been elaborated for PROs.

A prospective cohort study assessing clinical referral management & workforce allocation within a UK regional medical genetics service.

Ensuring patient access to genomic information in the face of increasing demand requires clinicians to develop innovative ways of working. This paper presents the first empirical prospective observational cohort study of UK multi-disciplinary genetic service delivery. It describes and explores collaborative working practices including the utilisation and role of clinical geneticists and non-medical genetic counsellors. Six hundred and fifty new patients referred to a regional genetics service were tracked through 850 clinical contacts until discharge. Referral decisions regarding allocation of lead health professional assigned to the case were monitored, including the use of initial clinical contact guidelines. Significant differences were found in the cases led by genetic counsellors and those led by clinical geneticists. Around a sixth, 16.8% (109/650) of referrals were dealt with by a letter back to the referrer or re-directed to another service provider and 14.8% (80/541) of the remaining patients chose not to schedule an appointment. Of the remaining 461 patients, genetic counsellors were allocated as lead health professional for 46.2% (213/461). A further 61 patients did not attend. Of those who did, 86.3% (345/400) were discharged after one or two appointments. Genetic counsellors contributed to 95% (784/825) of total patient contacts. They provided 93.7% (395/432) of initial contacts and 26.8% (106/395) of patients were discharged at that point. The information from this study informed a planned service re-design. More research is needed to assess the effectiveness and efficiency of different models of collaborative multi-disciplinary working within genetics services.

Defining therapeutic benefit for people with schizophrenia: focus on negative symptoms.

Schizophrenia is a complex, heterogeneous, multidimensional disorder within which negative symptoms are a significant and disabling feature. Whilst there is no established treatment for these symptoms, some pharmacological and psychosocial interventions have shown promise and this is an active area of research. Despite the effort to identify effective interventions, as yet there is no broadly accepted definition of therapeutic success. This article reviews concepts of clinical relevance and reports on a consensus conference whose goal was to apply these concepts to the treatment of negative symptoms. A number of key issues were identified and discussed including: assessment of specific negative symptom domains; defining response and remission for negative symptoms; assessment of functional outcomes; measurement of outcomes within clinical trials; and the assessment of duration/persistence of a response. The group reached a definition of therapeutic success using an achieved threshold of function that persisted over time. Recommendations were agreed upon with respect to: assessment of negative symptom domains of apathy-avolition and deficit of expression symptoms; thresholds for response and remission of negative symptoms based on level of symptomatology; assessing multiple domains of function including social occupation, activities of daily living, and socialization; the need for clinical trial data to include rate of change over time and converging sources of evidence; use of clinician, patient and caregiver perspectives to assess success; and the need for establishing criteria for the persistence of therapeutic benefit. A consensus statement and associated research criteria are offered as an initial step towards developing broad agreement regarding outcomes of negative symptoms treatment.

Abnormal stimulus-response intensity functions in posttraumatic stress disorder: an electrophysiological investigation.

OBJECTIVE: The purpose of this study was to explore the relationship between combat-related posttraumatic stress disorder (PTSD) and specific augmentation versus reduction patterns for the N100 and P200 components of auditory event-related potentials evoked by tones of increasing intensity. METHOD: Event-related potentials of subjects with PTSD (N=36), subjects with no psychopathology (N=20), subjects with major depression but no PTSD (N=10), and subjects with a history of chronic alcohol abuse but no PTSD (N=8) were recorded. Brain responses were evoked by a 2000-Hz tone presented in intensity blocks of 65, 72.5, 80, 87.5, and 95 dB (SPL). RESULTS: Evoked data from five PTSD subjects were of poor quality and excluded from further analyses. For all but one subject with no psychopathology and for all subjects with a history of alcohol abuse or major depression (but no PTSD), the Cz amplitude of the P200 response component showed augmentation as a nearly linear function of tone intensity. As a group, subjects with PTSD showed no such increase in P200 response magnitude. Examination of the data from individual subjects with PTSD showed that 42% exhibited augmentation patterns similar to those seen for subjects in the comparison groups. However, 58% showed evidence of P200 reduction, with the response to the loudest tone being smaller than responses to tones of intermediate intensity. CONCLUSIONS: The data suggest that there is a significant subgroup of patients with combat-related PTSD who enter into a state of protective inhibition at relatively low stimulus intensities. It is hypothesized that this is an appropriate adaptive mechanism for these subjects rather than an indication of a core neurobiological abnormality.

The role of human cognitive neuroscience in drug discovery for the dementias.

Cognitive dysfunction characterizes all the various forms of dementia. Evidence is accumulating that all of the progressive neurodegenerative dementias, such as Alzheimer's disease (AD), are preceded by years, if not decades, of pathological cognitive decline. The limited effectiveness of the four current medications registered for AD together with the failure of dozens of programmes over the last decade has influenced the decision to evaluate treatment at earlier stages of the disease; even before any cognitive symptoms have appeared. However, it has to be acknowledged that treating mild cognitive impairment (MCI) as a prodrome for AD has also had very limited success. Nonetheless a more important problem in MCI research, and dementia in general, has to be laid at the door of the limited effectiveness of the cognitive tests employed. This problem will become even more severe for the latest research direction of treating preclinical AD because such individuals will have levels of cognitive abilities which are in the normal range; and thus many of the scales currently used in dementia research will not be sufficiently demanding to identify change over time. This paper reviews and discusses the methodology and instruments available for research and clinical practice in this major area; with a focus on the challenges involved in test selection and evaluation.