A proposed solution to the base rate problem in the kappa statistic.

Because it corrects for chance agreement, kappa (kappa) is a useful statistic for calculating interrater concordance. However, kappa has been criticized because its computed value is a function not only of sensitivity and specificity, but also the prevalence, or base rate, of the illness of interest in the particular population under study. For example, it has been shown for a hypothetical case in which sensitivity and specificity remain constant at .95 each, that kappa falls from .81 to .14 when the prevalence drops from 50% to 1%. Thus, differing values of kappa may be entirely due to differences in prevalence. Calculation of agreement presents different problems depending on whether one is studying reliability or validity. We discuss quantification of agreement in the pure validity case, the pure reliability case, and those studies that fall somewhere between. As a way of minimizing the base rate problem, we propose a statistic for the quantification of agreement (the Y statistic), which can be related to kappa but which is completely independent of prevalence in the case of validity studies and relatively so in the case of reliability.

A comparison of two diagnostic methods. Clinical ICD diagnoses vs DSM-III and Research Diagnostic Criteria using the Diagnostic Interview Schedule (version 2).

In the context of a seven-year follow-up study, 171 former psychiatric inpatients and 158 subjects from the general population were interviewed twice, first with the German version of the Diagnostic Interview Schedule (DIS) (version 2), and second with a clinical interview using the Arbeitsgemeinschaft für Methodik und Dokumentation in der Psychiatrie (AMDP) checklist to assign a clinical International Classification of Diseases (eighth revision) (ICD-8) diagnosis, independent of the DIS. With the clinicians' ICD-8 diagnosis as a measure of the quality of the DIS, the results indicate a sufficiently high overall specificity and sensitivity of the DIS as a case-finding instrument in a general population survey, and a surprisingly high concordance of most DIS diagnostic classes with comparable ICD diagnoses. Only for panic disorders (possibly due to different symptom and time criteria) and schizophrenia (possibly due to the strict probe system, the dependence on self-reports, and time criteria) was low sensitivity found. Because there is no DIS diagnostic category comparable with ICD-8 unipolar affective psychosis, a meaningful comparison of this ICD category with Research Diagnostic Criteria and DSM-III was not possible.

A comparison of clinical and diagnostic interview schedule diagnoses. Physician reexamination of lay-interviewed cases in the general population.

We examined the level of agreement between diagnoses derived from data gathered by lay interviewers using the Diagnostic Interview Schedule (DIS) in a general population survey (the Epidemiologic Catchment Area project) and both DIS and clinical diagnoses made by psychiatrists. Overall percent agreement between the lay DIS and the psychiatrists clinical impression ranged from 79% to 96%. The chance-corrected concordance was .60 or better for eight of the 11 diagnoses. Specificities were all 90% or better. Sensitivities were lower, but lay results showed a bias for only two diagnoses: major depression was significantly underdiagnosed and obsessive illness was overdiagnosed. We compared the present results with those of previous studies from clinical settings. We explored possible reasons for disagreement and discussed the implications of the findings for psychiatric epidemiologic research.

The clinician's illusion.

There are several diseases, including schizophrenia, alcoholism, and opiate addiction, for which the long-term prognosis is subject to disagreement between clinicians and researchers and also among researchers. Part of this disagreement may be attributable to a difference in the populations they sample. The clinician samples the population currently suffering from the disease (a "prevalence" or census sample), while research samples tend to more nearly represent the population ever contracting the disease (an "incidence" sample). The clinician's sample is biased toward cases of long duration, since the probability that a case will appear in a prevalence sample is proportional to its duration, hence "the clinician's illusion." The statistical mechanism of this bias is illustrated and its consequences detailed. Other sources of sampling bias in clinical and research samples are briefly described and partial remedies are suggested.

Comparison of the lay Diagnostic Interview Schedule and a standardized psychiatric diagnosis. Experience in eastern Baltimore.

We studied DSM-III diagnoses made by the lay Diagnostic Interview Schedule (DIS) method in relation to a standardized DSM-III diagnosis by psychiatrists in the two-stage Baltimore Epidemiologic Catchment Area mental morbidity survey. Generally, prevalence estimates based on the DIS one-month diagnoses were significantly different from those based on the psychiatric diagnoses. Subjects identified as cases by each method were often different subjects. Measured in terms of kappa, the chance-corrected degree of agreement between the DIS and psychiatrists' one-month diagnoses was moderate for DSM-III alcohol-use disorder (abuse and dependence combined), and lower for other mental disorder categories. The unreliability of either the DIS or psychiatric diagnoses is one potential explanation for the observed disagreements. Others include the following: insufficient or inadequate information (on which to base a diagnosis); recency of disorder; incomplete criterion coverage; overinclusive DIS questions; and degree of reliance on subject symptom reports. Further study of the nature and sources of these discrepancies is underway. This work should produce a more complete understanding of obstacles to mental disorder case ascertainment by lay interview and clinical examination methods in the context of a field survey.

A reappraisal of the United Kingdom epidemic of fatal asthma. Can general mortality data implicate a therapeutic agent?

The 1960s epidemic of asthma deaths that affected young persons in England and Wales, as well as in other countries, was attributed to the effect of newly available pressurized aerosols containing sympathomimetic bronchodilators. The subsequent decision to ban the nonprescription sale of these agents in the United Kingdom represented a unique use of national and international mortality data. The application of such data for decisions about therapeutic agents has implications for the current rise of asthma deaths in New Zealand, for the recent United States regulatory action regarding the nonprescription sale of aerosolized bronchodilators, and for the appraisal of adverse reactions to other pharmaceutical substances. This article is concerned with the quality of the scientific evidence used to implicate bronchodilators in the 1960s epidemic, and also with the strengths and weaknesses of the ecologic studies on which the implication depended. After concluding that the causal link between asthma deaths and bronchodilators was not supported by satisfactory scientific evidence, we present new data and an alternative diagnostic-exchange hypothesis that may, in part, help explain the original association.

Changing epidemiology of tardive dyskinesia: an overview.

Dyskinesia is found significantly more often among neuroleptic-treated psychiatric patients than among non-neuroleptic-treated patients. The epidemiology of tardive dyskinesia is changing; its reported prevalence among neuroleptic-treated psychiatric inpatients has been progressively rising and has reached 25% during the past five years. The prevalence of persistent tardive dyskinesia that may be attributable to neuroleptics is about 13%. Tardive dyskinesia is not restricted to old, brain-damaged inpatients but also occurs with a noticeable frequency among younger patients, including outpatients, treated neuroleptics. Yet neuroleptics are the most effective available treatment for schizophrenia; hence, any drastic curtailment of their use in the treatment of chronic schizophrenic patients may not be justified. Cautious use of these drugs, along with intensified research on tardive dyskinesia is warranted.

Descriptive epidemiology of missed cases of phenylketonuria and congenital hypothyroidism.

We conducted a structured telephone survey of state public health laboratory directors of neonatal screening programs to determine the extent of the problem of missed cases of phenylketonuria (PKU) and congenital hypothyroidism. A total of 76 missed cases were reported--43 PKU and 33 congenital hypothyroidism. We looked at the following characteristics of the missed cases: the stage at which the miss occurred, which included specimen collection, laboratory procedures, or follow-up; the size of the program; the type of screening program; the age of the infant at the time of screening; and any legal action that resulted from the miss. The 76 missed cases probably represent an underascertainment of the true number, yet we believe that our data provide an overview of some of the problems associated with mass neonatal screening. There was one missed case of PKU for every 70 cases detected, and one missed case of congenital hypothyroidism for every 120 cases detected; in other words, two congenital hypothyroidism cases were missed for every 1 million infants screened. Regarding the stage of screening in which the miss occurred, 14% occurred during specimen collection, 45% during the laboratory procedures stage, 16% during follow-up, 11% were the result of biologic variation, and in 14% the stage could not be identified. We conclude that neonatal screening programs have been highly successful but that there may be additional safeguards to be developed, tested, and implemented when practical.

Assessing methods for causality assessment of suspected adverse drug reactions.

Reproducibility and validity as currently defined are inappropriate criteria for the evaluation of methods for causality assessment. Reproducibility leads to suppression rather than resolution of real disagreements and the method used to establish validity relies on the tarnished gold standard of expert opinion. We describe six alternative criteria that attempt to address a potential user's main concerns--the need to know whether to believe the results in general and in a particular case. These criteria focus on the internal structure of a method rather than its output. When we assessed the published methods by these criteria most of the methods failed most of the criteria. We believe that the problem and its solution lie at a fundamental level--real understanding of the true nature of causality assessment, which we suggest is an inherently subjective evaluation based on the multiple uncertainties that an assessor has about a case and not an objective attribute of the drug-event connection that can be determined from unambiguous evidence elicited in response to "operational questions".

Public and private hypotheses.

The uses of hypothesis in scientific medicine and in clinical medicine are broadly similar; but they differ in subtle but important logical details. The key distinction is that scientific medicine deals with broad ("public") hypotheses about entire populations; and the predominant problem of inference is statistics in the face of epistemological uncertainty. In contrast, clinical medicine deals with individual, tailor-made ("private") hypotheses; the uncertainty (e.g. the diagnosis or recurrence risk in relatives) is ontological, and the main feature of the analysis is probability. The same logical rules are at times mistakenly applied to both. Disregard of these subtleties may lead to paradoxes, contradictions, and at times diastrously false conclusion. The basic principles and distinctions are laid out with illustrations that are most readily provided from medical genetics, but apply to all branches of medicine.

The value and hazards of standardization in clinical epidemiologic research.

The statistical standardization of rates produces a single summary value that converts crude rates of occurrence into "standardized" rates that are adjusted for differences in the composition of compared populations. Although the process is well described in the epidemiologic literature and is regularly applied in comparisons of large populations, many investigators are not familiar with three important hazards that are magnified for the smaller groups studied in clinical epidemiologic research. This report contains a new "symmetrical" outline of the direct and indirect standardization processes, and an illustration of three pragmatic hazards: (1) Because the direct standardizing factor uses the observed stratum-specific rates, and because any stratum-specific rates that depend on small denominators may be misleading or unstable, the indirect method is preferred when the observed strata have small denominators. (2) Both the direct and indirect standardizing methods are highly vulnerable both to the choice of reference population and to the boundaries chosen when strata are demarcated or consolidated. The standardized rates can be altered dramatically according to differences in the stratum proportions of the reference population, or to distinctions produced when standardizing strata are consolidated. (3) If the stratum-specific rates and stratum proportions have different patterns of variation across the strata of the compared groups, the use of a single summary value--no matter what method of standardization is applied--may obscure cogent patterns of variation and significant differences in the stratum-specific rates. These hazards can be overcome if the studied group and the reference population are carefully compared for inconsistent variations in the stratum-specific rates and proportions before any standardizing procedure is applied. In many instances, the best approach may be to compare the unaltered stratum-specific rates, without standardization.

A critical evaluation of acute ozone epidemiology results.

Evidence from controlled human exposure studies demonstrates transient pulmonary functional responses to ozone exposure concentrations near the U.S. primary air quality standard of 0.120 ppm. There have been several recent efforts to document low concentration ozone effects in the epidemiological setting. Quantification of ozone effects under natural conditions of exposure can provide valuable information for risk assessment. However, results of epidemiological studies on acute ozone effects have had a limited role in decisions for air quality standards. This reflects difficulties in the quantitative interpretation of results that are to some extent inherent in the epidemiological approach, such as the difficulty of assessing individual exposures, failure to account for varying activity levels, and confounding by temporal covariates, e.g., temperature and pollen. However, in spite of the limitations of individual studies, when viewed as a group the epidemiological studies are consistent among themselves and with results from chamber studies.