Semiparametric methods for evaluating the covariate-specific predictiveness of continuous markers in matched case-control studies.

To assess the value of a continuous marker in predicting the risk of a disease, a graphical tool called the predictiveness curve has been proposed. It characterizes the marker's predictiveness, or capacity to risk stratify the population by displaying the distribution of risk endowed by the marker. Methods for making inference about the curve and for comparing curves in a general population have been developed. However, knowledge about a marker's performance in the general population only is not enough. Since a marker's effect on the risk model and its distribution can both differ across subpopulations, its predictiveness may vary when applied to different subpopulations. Moreover, information about the predictiveness of a marker conditional on baseline covariates is valuable for individual decision making about having the marker measured or not. Therefore, to fully realize the usefulness of a risk prediction marker, it is important to study its performance conditional on covariates. In this article, we propose semiparametric methods for estimating covariate-specific predictiveness curves for a continuous marker. Unmatched and matched case-control study designs are accommodated. We illustrate application of the methodology by evaluating serum creatinine as a predictor of risk of renal artery stenosis.

A parametric ROC model-based approach for evaluating the predictiveness of continuous markers in case-control studies.

The predictiveness curve shows the population distribution of risk endowed by a marker or risk prediction model. It provides a means for assessing the model's capacity for stratifying the population according to risk. Methods for making inference about the predictiveness curve have been developed using cross-sectional or cohort data. Here we consider inference based on case-control studies, which are far more common in practice. We investigate the relationship between the ROC curve and the predictiveness curve. Insights about their relationship provide alternative ROC interpretations for the predictiveness curve and for a previously proposed summary index of it. Next the relationship motivates ROC based methods for estimating the predictiveness curve. An important advantage of these methods over previously proposed methods is that they are rank invariant. In addition they provide a way of combining information across populations that have similar ROC curves but varying prevalence of the outcome. We apply the methods to prostate-specific antigen (PSA), a marker for predicting risk of prostate cancer.

Evaluating technologies for classification and prediction in medicine.

Modern technologies promise to provide new ways of diagnosing disease, detecting subclinical disease, predicting prognosis, selecting patient specific treatment, identifying subjects at risk for disease, and so forth. Advances in genomics, proteomics and imaging modalities in particular hold great potential for assisting with classification/prediction in medicine. Before a classifier can be adopted for routine use in health care, its classification accuracy must be determined. Standards for evaluating new clinical classifiers however, lag far behind the well established standards that exist for evaluating new clinical treatments. In this paper, we discuss a phased approach to developing a new classifier (or biomarker). It mirrors the internationally established phase 1-2-3 paradigm for therapeutic drugs. The defined phases lead to a logical sequence of studies for classifier development. We emphasize that evaluating classification accuracy is fundamentally different from simply establishing association with outcome. Therefore, study objectives and designs differ from the familiar methods of clinical trials. We discuss these briefly for each phase.Finally, we argue that classifier development requires some rethinking of traditional data analysis techniques. As an example we show that maximizing the likelihood function to fit a logistic regression model to multiple predictors, can yield a poor classifier. Instead we demonstrate that an approach that maximizes an alternative objective function characterizing classification accuracy performs better.

Effect of choice of reference equation on analysis of pulmonary function in cystic fibrosis patients.

Pulmonary function is an important measure of disease severity and prognosis in cystic fibrosis (CF). It is generally expressed as a percentage of a predicted value, calculated using regression equations derived from a reference population. A number of reference equations are in widespread use. The purposes of this study were to determine: 1) the extent to which, for a given absolute FEV(1) value, percent of predicted (PPFEV(1)) values vary when derived by different reference equations; and 2) whether these differences affect conclusions of longitudinal and cross-sectional analyses. Subjects were all Caucasians 6-18 years old in the 1990 Cystic Fibrosis Foundation Registry. We found clinically important discrepancies in PPFEV(1) when calculated by the methods of Dockery et al. [Am Rev Respir Dis 1983;128:405-412] and Wang et al. [Pediatr Pulmonol 1993;15:75-78] as compared to Knudson et al. [Am Rev Respir Dis 1983;127:725-734] or Polgar and Promadhat [Pulmonary Function Testing in Children 1971; Philadelphia: W.B. Saunders]. In longitudinal analyses, the choice of reference equation resulted in varying apparent rates of decline in FEV(1). For example, among subjects ages 12-14 years in 1990, the decline in PPFEV(1) from 1990-1995 varied between 2-11%, depending on the choice of reference equation. In cross-sectional analyses, the choice of reference equation affected the distribution of subjects classified as having mild, moderate, or severe lung disease. CF physicians should be aware of the impact of choice of reference equation in both clinical care and research.

Comparing disease screening tests when true disease status is ascertained only for screen positives.

Disease screening is a fundamental part of health care. To evaluate the accuracy of a new screening modality, ideally the results of the screening test are compared with those of a definitive diagnostic test in a set of study subjects. However, definitive diagnostic tests are often invasive and cannot be applied to subjects whose screening tests are negative for disease. For example, in cancer screening, the assessment of true disease status requires a biopsy sample, which for ethical reasons can only be obtained if a subject's screening test indicates presence of cancer. Although the absolute accuracy of screening tests cannot be evaluated in such circumstances, it is possible to compare the accuracies of screening tests. Specifically, using relative true positive rate (the ratio of the true positive rate of one test to another) and relative false positive rate (the ratio of the false positive rates of two tests) as measures of relative accuracy, we show that inference about relative accuracy can be made from such studies. Analogies with case-control studies can be drawn where inference about absolute risk cannot be made, but inference about relative risk can. In this paper, we develop a marginal regression analysis framework for making inference about relative accuracy when only screen positives are followed for true disease. In this context factors influencing the relative accuracies of tests can be evaluated. It is important to determine such factors in order to understand circumstances in which one test is preferable to another. The methods are applied to two cancer screening studies, one concerning the effect of race on screening for prostate cancer and the other concerning the effect of tumour grade on the detection of cervical cancer with cytology versus cervicography screening.

Time-dependent ROC curves for censored survival data and a diagnostic marker.

ROC curves are a popular method for displaying sensitivity and specificity of a continuous diagnostic marker, X, for a binary disease variable, D. However, many disease outcomes are time dependent, D(t), and ROC curves that vary as a function of time may be more appropriate. A common example of a time-dependent variable is vital status, where D(t) = 1 if a patient has died prior to time t and zero otherwise. We propose summarizing the discrimination potential of a marker X, measured at baseline (t = 0), by calculating ROC curves for cumulative disease or death incidence by time t, which we denote as ROC(t). A typical complexity with survival data is that observations may be censored. Two ROC curve estimators are proposed that can accommodate censored data. A simple estimator is based on using the Kaplan-Meier estimator for each possible subset X > c. However, this estimator does not guarantee the necessary condition that sensitivity and specificity are monotone in X. An alternative estimator that does guarantee monotonicity is based on a nearest neighbor estimator for the bivariate distribution function of (X, T), where T represents survival time (Akritas, M. J., 1994, Annals of Statistics 22, 1299-1327). We present an example where ROC(t) is used to compare a standard and a modified flow cytometry measurement for predicting survival after detection of breast cancer and an example where the ROC(t) curve displays the impact of modifying eligibility criteria for sample size and power in HIV prevention trials.

Comparisons of predictive values of binary medical diagnostic tests for paired designs.

Positive and negative predictive values of a diagnostic test are key clinically relevant measures of test accuracy. Surprisingly, statistical methods for comparing tests with regard to these parameters have not been available for the most common study design in which each test is applied to each study individual. In this paper, we propose a statistic for comparing the predictive values of two diagnostic tests using this paired study design. The proposed statistic is a score statistic derived from a marginal regression model and bears some relation to McNemar's statistic. As McNemar's statistic can be used to compare sensitivities and specificities of diagnostic tests, parameters that condition on disease status, our statistic can be considered as an analog of McNemar's test for the problem of comparing predictive values, parameters that condition on test outcome. We report on the results of a simulation study designed to examine the properties of this test under a variety of conditions. The method is illustrated with data from a study of methods for diagnosis of coronary artery disease.

An interpretation for the ROC curve and inference using GLM procedures.

The accuracy of a medical diagnostic test is often summarized in a receiver operating characteristic (ROC) curve. This paper puts forth an interpretation for each point on the ROC curve as being a conditional probability of a test result from a random diseased subject exceeding that from a random nondiseased subject. This interpretation gives rise to new methods for making inference about ROC curves. It is shown that inference can be achieved with binary regression techniques applied to indicator variables constructed from pairs of test results, one component of the pair being from a diseased subject and the other from a nondiseased subject. Within the generalized linear model (GLM) binary regression framework, ROC curves can be estimated, and we highlight a new semiparametric estimator. Covariate effects can also be evaluated with the GLM models. The methodology is applied to a pancreatic cancer dataset where we use the regression framework to compare two different serum biomarkers. Asymptotic distribution theory is developed to facilitate inference and to provide insight into factors influencing variability of estimated model parameters.

Statistical description of interrater variability in ordinal ratings.

Ordinal categorical assessments are common in medical practice and in research. Variability in such measurements amongst raters making the assessments can be problematic. In this paper we consider how such variability can be described statistically. We review three current approaches, including kappa-type statistics, loglinear models for agreement, and latent class agreement models, and discuss their limitations. We present a new graphical approach to describing interrater variability that involves a simple frequency distribution display of the category probabilities. The method enables description of interrater variability when raters are a random sample from some population as opposed to the traditional setting in which only a few selected raters provide assessments. Advantages of this approach relative to current approaches include the following: (1) it provides a simple visual summary of the rating data, (2) description is closely linked to familiar methods for describing variability in continuous measurements, (3) interpretation is straightforward, and (4) a large sample of raters can be accommodated with ease. We illustrate the method on simulated ordinal data representing radiologists' ratings of mammography images and on rating data from a national image reading study of mammography screening.

Combining diagnostic test results to increase accuracy.

When multiple diagnostic tests are performed on an individual or multiple disease markers are available it may be possible to combine the information to diagnose disease. We consider how to choose linear combinations of markers in order to optimize diagnostic accuracy. The accuracy index to be maximized is the area or partial area under the receiver operating characteristic (ROC) curve. We propose a distribution-free rank-based approach for optimizing the area under the ROC curve and compare it with logistic regression and with classic linear discriminant analysis (LDA). It has been shown that the latter method optimizes the area under the ROC curve when test results have a multivariate normal distribution for diseased and non-diseased populations. Simulation studies suggest that the proposed non-parametric method is efficient when data are multivariate normal.The distribution-free method is generalized to a smooth distribution-free approach to: (i) accommodate some reasonable smoothness assumptions; (ii) incorporate covariate effects; and (iii) yield optimized partial areas under the ROC curve. This latter feature is particularly important since it allows one to focus on a region of the ROC curve which is of most relevance to clinical practice. Neither logistic regression nor LDA necessarily maximize partial areas. The approaches are illustrated on two cancer datasets, one involving serum antigen markers for pancreatic cancer and the other involving longitudinal prostate specific antigen data.

Using a combination of reference tests to assess the accuracy of a new diagnostic test.

Often the accuracy of a new diagnostic test must be assessed when a perfect gold standard does not exist. Use of an imperfect reference test biases accuracy estimates of the new test. This paper reviews existing approaches to this problem including discrepant resolution and latent class analysis. Deficiencies with these approaches are identified. A new approach is proposed that combines the results of several imperfect reference tests to define a better reference standard. We call this the composite reference standard (CRS). Using the CRS, accuracy can be assessed using multi-stage sampling designs. Maximum likelihood estimates of accuracy and expressions for the variance of sensitivity and specificity are provided. Data from clinical literature on the detection of Chlamydia trachomatis are used to illustrate and compare the different approaches. Advantages of the CRS relative to other approaches include that the CRS is explicitly defined, does not depend on the results of the new test under investigation, and is easy to interpret.

Estimating and comparing univariate associations with application to the prediction of adult obesity.

Studies examining the association between an outcome variable and multiple predictors are common in medical research. Examples include epidemiologic studies of risk factors for disease and clinical studies of prognostic indicators for diseased subjects. This paper is concerned with the assessment of the associations between the outcome and each predictor separately, the so-called univariate associations. Comparisons between predictors in regards to the strengths of their association with the outcome are considered. We show that though such comparisons cannot be made with standard techniques, they can be made using an algorithm which performs all of the univariate analyses simultaneously. This is accomplished with a non-standard application of generalized estimating equation methods. Comparisons of univariate associations are shown to be the key analyses of interest in a retrospective longitudinal study of childhood predictors of adult obesity. We illustrate the methodology on data from this study.

Intermittent administration of inhaled tobramycin in patients with cystic fibrosis. Cystic Fibrosis Inhaled Tobramycin Study Group.

BACKGROUND AND METHODS: We conducted two multicenter, double-blind, placebo-controlled trials of intermittent administration of inhaled tobramycin in patients with cystic fibrosis and Pseudomonas aeruginosa infection. A total of 520 patients (mean age, 21 years) were randomly assigned to receive either 300 mg of inhaled tobramycin or placebo twice daily for four weeks, followed by four weeks with no study drug. Patients received treatment or placebo in three on-off cycles for a total of 24 weeks. The end points included pulmonary function, the density of P. aeruginosa in sputum, and hospitalization. RESULTS: The patients treated with inhaled tobramycin had an average increase in forced expiratory volume in one second (FEV1) of 10 percent at week 20 as compared with week 0, whereas the patients receiving placebo had a 2 percent decline in FEV1 (P<0.001). In the tobramycin group, the density of P. aeruginosa decreased by an average of 0.8 log10 colony-forming units (CFU) per gram of expectorated sputum from week 0 to week 20, as compared with an increase of 0.3 log10 CFU per gram in the placebo group (P<0.001). The patients in the tobramycin group were 26 percent (95 percent confidence interval, 2 to 43 percent) less likely to be hospitalized than those in the placebo group. Inhaled tobramycin was not associated with detectable ototoxic or nephrotoxic effects or with accumulation of the drug in serum. The proportion of patients with P. aeruginosa isolates for which the minimal inhibitory concentration of tobramycin was 8 microg per milliliter or higher increased from 25 percent at week 0 to 32 percent at week 24 in the tobramycin group, as compared with a decrease from 20 percent at week 0 to 17 percent at week 24 in the placebo group. CONCLUSIONS: In a 24-week study of patients with cystic fibrosis, intermittent administration of inhaled tobramycin was well tolerated and improved pulmonary function, decreased the density of P. aeruginosa in sputum, and decreased the risk of hospitalization.

Prediction using partly conditional time-varying coefficients regression models.

Data collected longitudinally in time provide the opportunity to develop predictive models of future observations given current data for an individual. Such models may be of particular value in defining individuals at high risk and thereby in suggesting subgroups for targeting of prevention intervention research efforts. In this paper, we propose a method for estimating predictive functions. The method uses an extension of the marginal regression analysis methods of Liang and Zeger (1986, Biometrika 73, 13-22) and is implemented using simple estimating equations. A key feature of the models is that regression coefficients are modelled as smooth functions of the times both at and for prediction. Data from a study of obesity in childhood and early adulthood is used to demonstrate the methodology. Criteria for defining individuals to be at high risk can be defined on the basis of estimated predictive functions. We suggest methods for evaluating the diagnostic accuracy (sensitivity and specificity) of such rules using cross-validation. The method holds promise as a robust and technically easy way of evaluating information about future prognosis that may be gleaned from a patient's current and past clinical status.

Effects of repetitive use and cleaning techniques of disposable jet nebulizers on aerosol generation.

STUDY OBJECTIVE: Patients with cystic fibrosis use disposable jet nebulizers for the self-administration of antibiotics, DNase, and bronchodilators several times per day. Most patients elect to reuse their disposable nebulizers. The purpose of this study was to determine if significant changes in particle size distribution or output (mL/min) occurred with reuse. DESIGN: In vitro studies were performed using four disposable models and one durable jet nebulizer for up to 100 runs; measurements of particle size and output were obtained at 10 run intervals, using saline solution alone, tobramycin, gentamicin, or a mixture of albuterol and cromolyn. Particle size determinations were made with a laser diffraction analyzer. RESULTS: There was no significant difference between the baseline performance of the four disposable models and the durable Pari LC, when measuring particle size distribution of the aerosol; the Pari LC had an output rate two to three times higher than the four disposable models. For each of the four solutes tested, there was no clinically significant change in performance for up to 100 cycles, when the nebulizers were properly cleaned between uses. Unwashed units containing tobramycin started to fail by 40 runs. CONCLUSIONS: When properly maintained, there was no trend of deterioration of performance with repeated use of disposable nebulizers. Microbial contamination was not addressed in this study and must be considered prior to recommendations for the reuse of disposable nebulizers.

Regression modelling of diagnostic likelihood ratios for the evaluation of medical diagnostic tests.

The use of diagnostic likelihood ratios has been advocated in the epidemiologic literature for the past decade. Diagnostic likelihood ratios provide valuable information about the predictive properties of a diagnostic test while having the attractive feature of being independent of the prevalence of disease in the study population. We propose a new regression method that allows for direct assessment of covariate effects on likelihood ratios for binary diagnostic tests. This may be particularly useful in assessing how factors that are under the control of the clinician can be altered to maximize the predictive ability of the test. Similarly, patient characteristics that influence the ability of the test to discriminate between diseased and nondiseased subjects may be identified using the regression model. The regression method is flexible in that it can accommodate clustered data arising from a variety of study designs. We illustrate the method with data from an audiology study.

Three approaches to regression analysis of receiver operating characteristic curves for continuous test results.

The accuracy of a medical diagnostic test is typically summarized by the sensitivity and specificity when the test result is dichotomous. Receiver operating characteristic (ROC) curves are measures of test accuracy that are used when test results are continuous and are considered the analogs of sensitivity and specificity for continuous tests. ROC regression analysis allows one to evaluate effects of factors that may influence test accuracy. Such factors might include characteristics of study subjects or operating conditions for the test. Unfortunately, regression analysis methods for ROC curves are not well developed and methods that do exist have received little use to date. In this paper, we propose and compare three very different regression analysis methods. Two are modifications of methods previously proposed for radiology settings. The third is a special case of a general method recently proposed by us. The three approaches are compared with regard to settings in which they can be applied and distributional assumptions they require. In the setting where test results are normally distributed, we elucidate the correspondence between regression parameters in the different models. The methods are applied to simulated data and to data from a study of a new diagnostic test for hearing impairment. It is hoped that the presentation in this paper will both encourage the use of regression analysis for evaluating diagnostic tests and help guide the choice of the most appropriate regression analysis approach in applications.

Early adiposity rebound and the risk of adult obesity.

OBJECTIVE: At 5 to 6 years of age, body fatness normally declines to a minimum, a point called adiposity rebound (AR), before increasing again into adulthood. We determined whether a younger age at AR was associated with an increased risk of adult obesity and whether this risk was independent of fatness at AR and parent obesity. DESIGN: A retrospective cohort study using lifelong height and weight measurements recorded in outpatient medical records. SETTING: Group Health Cooperative of Puget Sound (GHC), a health maintenance organization based in Seattle, Washington. PARTICIPANTS: All 390 GHC members (and their parents) born at GHC between January 1, 1965, and January 1, 1971, who had at least one recorded adult height and weight measurement plus two visits with recorded height and weight measurements in each of three age intervals: 1.5 to 4, 4 to 8, and 8 to 16 years. MAIN OUTCOME MEASURES: We calculated the mean body mass index (BMI) of each subject during young adulthood (age 21 to 29 years) and the BMI of the parents when each subject was 1.5 years of age. Adult obesity was defined as a BMI >/=27.8 for males and >/=27. 3 for females. Curves were fit to each subject's BMI values between ages 1.5 and 16 years, and the age and BMI at AR were calculated from these curves. Subjects were divided into tertiles of age at AR (early, middle, and late), BMI at AR, and parent BMI (heavy, medium, and lean). RESULTS: The mean age at AR was 5.5 years, and 15% of the cohort was obese in young adulthood. Adult obesity rates were higher in those with early versus late AR (25% vs 5%), those who were heavy versus lean at AR (24% vs 4%), those with heavy versus lean mothers (25% vs 5%), and those with heavy versus lean fathers (21% vs 5%). After adjusting for parent BMI and BMI at AR, the odds ratio for adult obesity associated with early versus late AR was 6.0 (95% CI, 1.3-26.6). CONCLUSION: An early AR is associated with an increased risk of adult obesity independent of parent obesity and the BMI at AR. Future research should examine the biological and behavioral determinants of AR.