On point estimation of the abnormality of a Mahalanobis index.

Mahalanobis distance may be used as a measure of the disparity between an individual's profile of scores and the average profile of a population of controls. The degree to which the individual's profile is unusual can then be equated to the proportion of the population who would have a larger Mahalanobis distance than the individual. Several estimators of this proportion are examined. These include plug-in maximum likelihood estimators, medians, the posterior mean from a Bayesian probability matching prior, an estimator derived from a Taylor expansion, and two forms of polynomial approximation, one based on Bernstein polynomial and one on a quadrature method. Simulations show that some estimators, including the commonly-used plug-in maximum likelihood estimators, can have substantial bias for small or moderate sample sizes. The polynomial approximations yield estimators that have low bias, with the quadrature method marginally to be preferred over Bernstein polynomials. However, the polynomial estimators sometimes yield infeasible estimates that are outside the 0-1 range. While none of the estimators are perfectly unbiased, the median estimators match their definition; in simulations their estimates of the proportion have a median error close to zero. The standard median estimator can give unrealistically small estimates (including 0) and an adjustment is proposed that ensures estimates are always credible. This latter estimator has much to recommend it when unbiasedness is not of paramount importance, while the quadrature method is recommended when bias is the dominant issue.

Automated biosurveillance data from England and Wales, 1991-2011.

Outbreak detection systems for use with very large multiple surveillance databases must be suited both to the data available and to the requirements of full automation. To inform the development of more effective outbreak detection algorithms, we analyzed 20 years of data (1991-2011) from a large laboratory surveillance database used for outbreak detection in England and Wales. The data relate to 3,303 distinct types of infectious pathogens, with a frequency range spanning 6 orders of magnitude. Several hundred organism types were reported each week. We describe the diversity of seasonal patterns, trends, artifacts, and extra-Poisson variability to which an effective multiple laboratory-based outbreak detection system must adjust. We provide empirical information to guide the selection of simple statistical models for automated surveillance of multiple organisms, in the light of the key requirements of such outbreak detection systems, namely, robustness, flexibility, and sensitivity.

Point and interval estimates of effect sizes for the case-controls design in neuropsychology: rationale, methods, implementations, and proposed reporting standards.

It is increasingly common for group studies in neuropsychology to report effect sizes. In contrast this is rarely done in single-case studies (at least in those studies that employ a case-controls design). The present paper sets out the advantages of reporting effect sizes, derives suitable effect size indexes for use in single-case studies, and develops methods of supplementing point estimates of effect sizes with interval estimates. Computer programs that implement existing classical and Bayesian inferential methods for the single case (as developed by Crawford, Garthwaite, Howell, and colleagues) are upgraded to provide these point and interval estimates. The upgraded programs can be downloaded from www.abdn.ac.uk/~psy086/dept/Single_Case_Effect_Sizes.htm.

Inferential methods for comparing two single cases.

In neuropsychological single-case studies, it is not uncommon for researchers to compare the scores of two single cases. Classical (and Bayesian) statistical methods are developed for such problems, which, unlike existing methods, refer the scores of the two single cases to a control sample. These methods allow researchers to compare two cases' scores, with or without allowing for the effects of covariates. The methods provide a hypothesis test (one- or two-tailed), point and interval estimates of the effect size of the difference, and point and interval estimates of the percentage of pairs of controls that will exhibit larger differences than the cases. Monte Carlo simulations demonstrate that the statistical theory underlying the methods is sound and that the methods are robust in the face of departures from normality. The methods have been implemented in computer programs, and these are described and made available (to download, go to http://www.abdn.ac.uk/~psy086/dept/Compare_Two_Cases.htm).

A convenient method of obtaining percentile norms and accompanying interval estimates for self-report mood scales (DASS, DASS-21, HADS, PANAS, and sAD).

BACKGROUND: A series of recent papers have reported normative data from the general adult population for commonly used self-report mood scales. AIMS: To bring together and supplement these data in order to provide a convenient means of obtaining percentile norms for the mood scales. METHOD: A computer program was developed that provides point and interval estimates of the percentile rank corresponding to raw scores on the various self-report scales. RESULTS: The program can be used to obtain point and interval estimates of the percentile rank of an individual's raw scores on the DASS, DASS-21, HADS, PANAS, and sAD mood scales, based on normative sample sizes ranging from 758 to 3822. The interval estimates can be obtained using either classical or Bayesian methods as preferred. CONCLUSION: The computer program (which can be downloaded at www.abdn.ac.uk/~psy086/dept/MoodScore.htm) provides a convenient and reliable means of supplementing existing cut-off scores for self-report mood scales.

On the reliability and standard errors of measurement of contrast measures from the D-KEFS.

A formula for the reliability of difference scores was used to estimate the reliability of Delis-Kaplan Executive Function System (D-KEFS; Delis et al., 2001) contrast measures from the reliabilities and correlations of their components. In turn these reliabilities were used to calculate standard errors of measurement. The majority of contrast measures had low reliabilities: of the 51 reliability coefficients calculated in the present study, none exceeded 0.7 and hence all failed to meet any of the criteria for acceptable reliability proposed by various experts in psychological measurement. The mean reliability of the contrast scores was 0.27, the median reliability was 0.30. The standard errors of measurement were large and, in many cases, equaled or were only marginally smaller than the contrast scores' standard deviations. The results suggest that, at present, D-KEFS contrast measures should not be used in neuropsychological decision making.

On the "optimal" size for normative samples in neuropsychology: capturing the uncertainty when normative data are used to quantify the standing of a neuropsychological test score.

Bridges and Holler (2007) have provided a useful reminder that normative data are fallible. Unfortunately, however, their paper misleads neuropsychologists as to the nature and extent of the problem. We show that the uncertainty attached to the estimated z score and percentile rank of a given raw score is much larger than they report and that it varies as a function of the extremity of the raw score. Methods for quantifying the uncertainty associated with normative data are described and used to illustrate the issues involved. A computer program is provided that, on entry of a normative sample mean, standard deviation, and sample size, provides point and interval estimates of percentiles and z scores for raw scores referred to these normative data. The methods and program provide neuropsychologists with a means of evaluating the adequacy of existing norms and will be useful for those planning normative studies.

Comparison of a single case to a control or normative sample in neuropsychology: development of a Bayesian approach.

Frequentist methods are available for comparison of a patient's test score (or score difference) to a control or normative sample; these methods also provide a point estimate of the percentage of the population that would obtain a more extreme score (or score difference) and, for some problems, an accompanying interval estimate (i.e., confidence limits) on this percentage. In the present paper we develop a Bayesian approach to these problems. Despite the very different approaches, the Bayesian and frequentist methods yield equivalent point and interval estimates when (a) a case's score is compared to that of a control sample, and (b) when the raw (i.e., unstandardized) difference between a case's scores on two tasks are compared to the differences in controls. In contrast, the two approaches differ with regard to point estimates of the abnormality of the difference between a case's standardized scores. The Bayesian method for standardized differences has the advantages that (a) it can directly evaluate the probability that a control will obtain a more extreme difference score, (b) it appropriately incorporates error in estimating the standard deviations of the tasks from which the patient's difference score is derived, and (c) it provides a credible interval for the abnormality of the difference between an individual's standardized scores; this latter problem has failed to succumb to frequentist methods. Computer programs that implement the Bayesian methods are described and made available.

Using regression equations built from summary data in the neuropsychological assessment of the individual case.

Regression equations have many useful roles in neuropsychological assessment. This article is based on the premise that there is a large reservoir of published data that could be used to build regression equations; these equations could then be used to test a wide variety of hypotheses concerning the functioning of individual cases. This resource is currently underused because (a) not all neuropsychologists are aware that equations can be built with only basic summary data for a sample and (b) the computations involved are tedious and prone to error. To overcome these barriers, the authors set out the steps required to build regression equations from sample summary statistics and the further steps required to compute the associated statistics for drawing inferences concerning an individual case. The authors also develop, describe, and make available computer programs that implement the methods. Although caveats attach to the use of the methods, these need to be balanced against pragmatic considerations and against the alternative of either entirely ignoring a pertinent data set or using it informally to provide a clinical "guesstimate."

Estimating the percentage of the population with abnormally low scores (or abnormally large score differences) on standardized neuropsychological test batteries: a generic method with applications.

Information on the rarity or abnormality of an individual's test scores (or test score differences) is fundamental in interpreting the results of a neuropsychological assessment. If a standardized battery of tests is administered, the question arises as to what percentage of the healthy population would be expected to exhibit one or more abnormally low test scores (and, in general, j or more abnormally low scores). Similar issues arise when the concern is with the number of abnormal pairwise differences between an individual's scores on the battery, or when an individual's scores on each component of the battery are compared with the individual's mean score. A generic Monte Carlo simulation method for tackling such problems is described (it requires only that the matrix of correlations between tests be available) and is contrasted with the use of binomial probabilities. The method is then applied to Index scores for the Wechsler Adult Intelligence Scale--Third Edition (WAIS-III; D. Wechsler, 1997) and Wechsler Intelligence Scale for Children--Fourth Edition (WISC-IV; D. Wechsler, 2003). Three computer programs that implement the methods are made available.

Detecting dissociations in single-case studies: Type I errors, statistical power and the classical versus strong distinction.

Dissociations observed in single-case studies play an important role in building and testing theory in neuropsychology; therefore the criteria used to identify their presence should be subjected to empirical scrutiny. Extending work on classical dissociations, Monte Carlo simulation is used to examine the Type I error rate for two methods of detecting strong dissociations. When a Type I error was defined as misclassifying a healthy control, error rates were low for both methods. When Type I errors were defined as misclassifying patients with strictly equivalent deficits on two tasks, error rates for strong dissociations were high for the "conventional" criteria and were very high when cases misclassified as exhibiting either form of dissociation (strong or classical) were combined (maximum = 55.1%). The power to detect strong and classical dissociations was generally low-to-moderate, but was moderate-to-high in most scenarios when power was defined as the ability to detect either form of dissociation. In most scenarios patients with strong dissociations were more likely to be classified as exhibiting classical dissociations. The results question the practical utility of the distinction between strong and classical dissociations regardless of the criteria employed to test for their presence.

Testing for a deficit in single-case studies: effects of departures from normality.

In neuropsychological single-case research inferences concerning a patient's cognitive status are often based on referring the patient's test score to those obtained from a modestly sized control sample. Two methods of testing for a deficit (z and a method proposed by Crawford and Howell [Crawford, J. R. & Howell, D. C. (1998). Comparing an individual's test score against norms derived from small samples. The Clinical Neuropsychologist, 12, 482-486]) both assume the control distribution is normal but this assumption will often be violated in practice. Monte Carlo simulation was employed to study the effects of leptokurtosis and the combination of skew and leptokurtosis on the Type I error rates for these two methods. For Crawford and Howell's method, leptokurtosis produced only a modest inflation of the Type I error rate when the control sample N was small-to-modest in size and error rates were lower than the specified rates at larger N. In contrast, the combination of leptokurtosis and skew produced marked inflation of error rates for small Ns. With a specified error rate of 5%, actual error rates as high as 14.31% and 9.96% were observed for z and Crawford and Howell's method respectively. Potential solutions to the problem of non-normal data are evaluated.

Comparing patients' predicted test scores from a regression equation with their obtained scores: a significance test and point estimate of abnormality with accompanying confidence limits.

In contrast to the standard use of regression, in which an individual's score on the dependent variable is unknown, neuropsychologists are often interested in comparing a predicted score with a known obtained score. Existing inferential methods use the standard error for a new case (s-subN+1) to provide confidence limits on a predicted score and hence are tailored to the standard usage. However, s-subN+1 can be used to test whether the discrepancy between a patient's predicted and obtained scores was drawn from the distribution of discrepancies in a control population. This method simultaneously provides a point estimate of the percentage of the control population that would exhibit a larger discrepancy. A method for obtaining confidence limits on this percentage is also developed. These methods can be used with existing regression equations and are particularly useful when the sample used to generate a regression equation is modest in size. Monte Carlo simulations confirm the validity of the methods, and computer programs that implement them are described and made available.

Evaluating the Contributions of Individual Variables to a Quadratic Form.

Quadratic forms capture multivariate information in a single number, making them useful, for example, in hypothesis testing. When a quadratic form is large and hence interesting, it might be informative to partition the quadratic form into contributions of individual variables. In this paper it is argued that meaningful partitions can be formed, though the precise partition that is determined will depend on the criterion used to select it. An intuitively reasonable criterion is proposed and the partition to which it leads is determined. The partition is based on a transformation that maximises the sum of the correlations between individual variables and the variables to which they transform under a constraint. Properties of the partition, including optimality properties, are examined. The contributions of individual variables to a quadratic form are less clear-cut when variables are collinear, and forming new variables through rotation can lead to greater transparency. The transformation is adapted so that it has an invariance property under such rotation, whereby the assessed contributions are unchanged for variables that the rotation does not affect directly. Application of the partition to Hotelling's one- and two-sample test statistics, Mahalanobis distance and discriminant analysis is described and illustrated through examples. It is shown that bootstrap confidence intervals for the contributions of individual variables to a partition are readily obtained.

Comparison of Statistical Algorithms for the Detection of Infectious Disease Outbreaks in Large Multiple Surveillance Systems.

A large-scale multiple surveillance system for infectious disease outbreaks has been in operation in England and Wales since the early 1990s. Changes to the statistical algorithm at the heart of the system were proposed and the purpose of this paper is to compare two new algorithms with the original algorithm. Test data to evaluate performance are created from weekly counts of the number of cases of each of more than 2000 diseases over a twenty-year period. The time series of each disease is separated into one series giving the baseline (background) disease incidence and a second series giving disease outbreaks. One series is shifted forward by twelve months and the two are then recombined, giving a realistic series in which it is known where outbreaks have been added. The metrics used to evaluate performance include a scoring rule that appropriately balances sensitivity against specificity and is sensitive to variation in probabilities near 1. In the context of disease surveillance, a scoring rule can be adapted to reflect the size of outbreaks and this was done. Results indicate that the two new algorithms are comparable to each other and better than the algorithm they were designed to replace.

Evaluation of criteria for classical dissociations in single-case studies by Monte Carlo simulation.

The conventional criteria for a classical dissociation in single-case studies require that a patient be impaired on one task and within normal limits on another. J. R. Crawford and P. H. Garthwaite (2005) proposed an additional criterion, namely, that the patient's (standardized) difference on the two tasks should differ from the distribution of differences in controls. Monte Carlo simulation was used to evaluate these criteria. When Type I errors were defined as falsely concluding that a control case exhibited a dissociation, error rates were high for the conventional criteria but low for Crawford and Garthwaite's criteria. When Type I error rates were defined as falsely concluding that a patient with equivalent deficits on the two tasks exhibited a dissociation, error rates were very high for the conventional criteria but acceptable for the latter criteria. These latter criteria were robust in the face of nonnormal control data. The power to detect classical dissociations was studied.

Testing for suspected impairments and dissociations in single-case studies in neuropsychology: evaluation of alternatives using monte carlo simulations and revised tests for dissociations.

In neuropsychological single-case studies, a patient is compared with a small control sample. Methods of testing for a deficit on Task X, or a significant difference between Tasks X and Y, either treat the control sample statistics as parameters (using z and zD) or use modified t tests. Monte Carlo simulations demonstrated that if z is used to test for a deficit, the Type I error rate is high for small control samples, whereas control of the error rate is essentially perfect for a modified t test. Simulations on tests for differences revealed that error rates were very high for zD. A new method of testing for a difference (the revised standardized difference test) achieved good control of the error rate, even with very small sample sizes. A computer program that implements this new test (and applies criteria to test for classical and strong dissociations) is made available.

Investigation of the single case in neuropsychology: confidence limits on the abnormality of test scores and test score differences.

Neuropsychologists often need to estimate the abnormality of an individual patient's test score, or test score discrepancies, when the normative or control sample against which the patient is compared is modest in size. Crawford and Howell [The Clinical Neuropsychologist 12 (1998) 482] and Crawford et al. [Journal of Clinical and Experimental Neuropsychology 20 (1998) 898] presented methods for obtaining point estimates of the abnormality of test scores and test score discrepancies in this situation. In the present study, we extend this work by developing methods of setting confidence limits on the estimates of abnormality. Although these limits can be used with data from normative or control samples of any size, they will be most useful when the sample sizes are modest. We also develop a method for obtaining point estimates and confidence limits on the abnormality of a discrepancy between a patient's mean score on k-tests and a test entering into that mean. Computer programs that implement the formulae for the confidence limits (and point estimates) are described and made available.

Statistical methods for single-case studies in neuropsychology: comparing the slope of a patient's regression line with those of a control sample.

Performance on some neuropsychological tests is best expressed as the slope of a regression line. Examples include the quantification of performance on tests designed to assess the accuracy of time estimation or distance estimation. The present paper presents methods for comparing a patient's performance with a control or normative sample when performance is expressed as slope. The methods test if there is a significant difference between a patient's slope and those obtained from controls, yield an estimate of the abnormality of the patient's slope, and provide confidence limits on the level of abnormality. The methods can be used with control samples of any size and will therefore be of particular relevance to single-case researchers. A method for comparing the difference between a patient's scores on two measures with the differences observed in controls is also described (one or both measures can be slopes). The methods require only summary statistics (rather than the raw data from the normative or control sample); it is hoped that this feature will encourage the development of norms for tasks that use slopes to quantify performance. Worked examples of the statistical methods are provided using neuropsychological data and a computer program (for PCs) that implements the methods is described and made available.

Wanted: fully operational definitions of dissociations in single-case studies.

In contrast to the careful consideration given to the issue of what we can infer from dissociations in single-case studies, the more basic question of how we decide whether a dissociation is present has been relatively neglected. Proposals are made for fully operational definitions of a deficit, classical and strong dissociations, and double dissociations. In developing these definitions it was assumed that they should be based on the use of inferential rather than descriptive statistical methods. The scope of these definitions is limited to typical single-case studies in which patients are compared to control samples of a modest size. The operational definition of a classical dissociation incorporates a requirement that a patient's performance on Task X is significantly different from Task Y, in addition to the "standard" requirement that the patient has a deficit on Task X and is within normal limits on Task Y. We ran a simulation to estimate the Type I error rates when the criteria for dissociations are applied and found these to be low (Type I errors were defined as identifying an individual from the control population as having a dissociation). The inferential methods for testing whether the various criteria are met make use of t-distributions. These methods are contrasted with the widespread use of z to test for a deficit or a difference between tasks. In the latter approach the statistics of the control sample are treated as parameters; this is not appropriate when, as is normally the case, the control sample size is modest in size.