Assessing the sources of unreliability (rater, subject, time-point) in a failed clinical trial using items of the Positive and Negative Syndrome Scale (PANSS).

BACKGROUND: Considering the increasing attention to the study of failed clinical trials, the goal of this study was to identify the sources of unreliability in a failed clinical trial by assessing scores on the Positive and Negative Syndrome Scale (PANSS). METHODS: This study is a substudy from a failed phase 2 double-blind, placebo-controlled trial of schizophrenia. Using the generalizability theory, this substudy assesses reliability on 3 conditions: raters, time points (PANSS evaluations, 1 week apart), subjects for 3 groups (placebo responders, placebo nonresponders, and treatment group). RESULTS: The placebo response rate was 40.07% (32/71). For all PANSS positive symptom items, the most variability was for raters (range, 33%-72%) for the placebo responders, 31% to 68% for the placebo nonresponders, and 29% to 60% for the treatment group. The variability of the interaction of rater and time point was the second source of unreliability, with an average of 12.28% compared to 12.00% for the placebo nonresponders and 10.00% for the treatment group. All items of the negative symptom subscale showed the most percent variability for raters, for all groups. For general psychopathology items (except preoccupation), raters accounted for the most variability in the scores for placebo responders with an average of 51.00% across items. A similar pattern was observed for the placebo nonresponders and for the treatment group; for the treatment group, the interaction between rater and time point accounted for the most variability for somatic concern and anxiety. CONCLUSIONS: Results confirm the efficacy of applying the generalizability theory to the estimation of reliability to identify a source of unreliability and provide evidence for the relationship between low reliability and failed trials. Findings can be used to guide data monitoring, rater training, and identification of PANSS items, which may require supplementary training.

A Developmental History of the Positive and Negative Syndrome Scale (PANSS).

Psychometric instruments are products of their time-Their designs and initial purposes are influenced and shaped by the contemporary treatment regimens, context, and cultural and conceptual biases of their developers. In this review article, the authors explore the history of the most influential schizophrenia research tools that have been created over the past several decades. The authors describe the scientific concepts, cultural influences, and challenges of past and present researchers as they strive to develop better assessment tools for schizophrenia. Starting with Moore's Scheme for the Quantitative Measurement of Abnormal Emotional Condition, developed in the early 1900s, and concluding with Kay, Fiszbein, and Opler's Positive and Negative Syndrome Scale, developed in the 80s, the authors describe several scales and illustrate how each scale led to and influenced the development of a later scale. The authors hope that a better understanding of schizophrenia assessment tool evolution and limitations will assist in the development of new instruments that better address the global needs for the evaluation, research, and treatment of psychosis.

Reliability of the Alzheimer's disease assessment scale (ADAS-Cog) in longitudinal studies.

BACKGROUND: Considering the scarcity of longitudinal assessments of reliability, there is need for a more precise understanding of cognitive decline in Alzheimer's Disease (AD). The primary goal was to assess longitudinal changes in inter-rater reliability, test retest reliability and internal consistency of scores of the ADAS-Cog. METHODS: 2,618 AD subjects were enrolled in seven randomized, double-blind, placebo-controlled, multicenter-trials from 1986 to 2009. Reliability, internal-consistency and cross-sectional analysis of ADAS-Cog and MMSE across seven visits were examined. RESULTS: Intra-class correlation (ICC) for ADAS-Cog was moderate to high supporting their reliability. Absolute Agreement ICCs 0.392 (Visit-7) to 0.806 (Visit-2) showed a progressive decrease in correlations across time. Item analysis revealed a decrease in item correlations, with the lowest correlations for Visit 7 for Commands (ICC=0.148), Comprehension (ICC=0.092), Spoken Language (ICC=0.044). DISCUSSION: Suitable assessment of AD treatments is maintained through accurate measurement of clinically significant outcomes. Targeted rater education ADAS-Cog items over-time can improve ability to administer and score the scale.

A rasch model to test the cross-cultural validity in the positive and negative syndrome scale (PANSS) across six geo-cultural groups.

BACKGROUND: The objective of this study was to examine the cross-cultural differences of the PANSS across six geo-cultural regions. The specific aims are (1) to examine measurement properties of the PANSS; and (2) to examine how each of the 30 items function across geo-cultural regions. METHODS: Data was obtained for 1,169 raters from 6 different regions: Eastern Asia (n = 202), India (n = 185), Northern Europe (n = 126), Russia & Ukraine (n = 197), Southern Europe (n = 162), United States (n = 297). A principle components analysis assessed unidimensionality of the subscales. Rasch rating scale analysis examined cross-cultural differences among each item of the PANSS. RESULTS: Lower item values reflects items in which raters often showed less variation in the scores; higher item values reflects items with more variation in the scores. Positive Subscale: Most regions found item P5 (Excitement) to be the most difficult item to score. Items varied in severity from -0.93 [item P6. Suspiciousness/persecution (USA) to 0.69 item P4. Excitement (Eastern Asia)]. Item P3 (Hallucinatory Behavior) was the easiest item to score for all geographical regions. Negative Subscale: The most difficult item to score for all regions is N7 (Stereotyped Thinking) with India showing the most difficulty Δ = 0.69, and Northern Europe and the United States showing the least difficulty Δ = 0.21, each. The second most difficult item for raters to score was N1 (Blunted Affect) for most countries including Southern Europe (Δ = 0.30), Eastern Asia (Δ = 0.28), Russia & Ukraine (Δ = 0.22) and India (Δ = 0.10). General Psychopathology: The most difficult item for raters to score for all regions is G4 (Tension) with difficulty levels ranging from Δ = 1.38 (India) to Δ = 0.72. CONCLUSIONS: There were significant differences in response to a number of items on the PANSS, possibly caused by a lack of equivalence between the original and translated versions, cultural differences among interpretation of items or scoring parameters. Knowing which items are problematic for various cultures can help guide PANSS training and make training specialized for specific geographical regions.