Inference on the overlap coefficient: The binormal approach and alternatives.

The overlap coefficient (OVL) measures the similarity between two distributions through the overlapping area of their distribution functions. Given its intuitive description and ease of visual representation by the straightforward depiction of the amount of overlap between the two corresponding histograms based on samples of measurements from each one of the two distributions, the development of accurate methods for confidence interval construction can be useful for applied researchers. The overlap coefficient has received scant attention in the literature since it lacks readily available software for its implementation, while inferential procedures that can cover the whole range of distributional scenarios for the two underlying distributions are missing. Such methods, both parametric and non-parametric are developed in this article, while R-code is provided for their implementation. Parametric approaches based on the binormal model show better performance and are appropriate for use in a wide range of distributional scenarios. Methods are assessed through a large simulation study and are illustrated using a dataset from a study on human immunodeficiency virus-related cognitive function assessment.

Statistical inference for the difference between two maximized Youden indices obtained from correlated biomarkers.

Currently, there is global interest in deriving new promising cancer biomarkers that could complement or substitute the conventional ones. Clinical decisions can often be based on the cutoff that corresponds to the maximized Youden index when maximum accuracy drives decisions. When more than one classification criteria are measured within the same individuals, correlated measurements arise. In this work, we propose hypothesis tests and confidence intervals for the comparison of two correlated receiver operating characteristic (ROC) curves in terms of their corresponding maximized Youden indices. We explore delta-based techniques under parametric assumptions, or power transformations. Nonparametric kernel-based methods are also examined. We evaluate our approaches through simulations and illustrate them using data from a metabolomic study referring to the detection of pancreatic cancer.

Construction of confidence intervals for the maximum of the Youden index and the corresponding cutoff point of a continuous biomarker.

Evaluation of the overall accuracy of biomarkers might be based on average measures of the sensitivity for all possible specificities -and vice versa- or equivalently the area under the receiver operating characteristic (ROC) curve that is typically used in such settings. In practice clinicians are in need of a cutoff point to determine whether intervention is required after establishing the utility of a continuous biomarker. The Youden index can serve both purposes as an overall index of a biomarker's accuracy, that also corresponds to an optimal, in terms of maximizing the Youden index, cutoff point that in turn can be utilized for decision making. In this paper, we provide new methods for constructing confidence intervals for both the Youden index and its corresponding cutoff point. We explore approaches based on the delta approximation under the normality assumption, as well as power transformations to normality and nonparametric kernel- and spline-based approaches. We compare our methods to existing techniques through simulations in terms of coverage and width. We then apply the proposed methods to serum-based markers of a prospective observational study involving diagnosis of late-onset sepsis in neonates.

Confidence intervals for differences between volumes under receiver operating characteristic surfaces (VUS) and generalized Youden indices (GYIs).

This article explores both existing and new methods for the construction of confidence intervals for differences of indices of diagnostic accuracy of competing pairs of biomarkers in three-class classification problems and fills the methodological gaps for both parametric and non-parametric approaches in the receiver operating characteristic surface framework. The most widely used such indices are the volume under the receiver operating characteristic surface and the generalized Youden index. We describe implementation of all methods and offer insight regarding the appropriateness of their use through a large simulation study with different distributional and sample size scenarios. Methods are illustrated using data from the Alzheimer's Disease Neuroimaging Initiative study, where assessment of cognitive function naturally results in a three-class classification setting.

Construction of joint confidence regions for the optimal true class fractions of Receiver Operating Characteristic (ROC) surfaces and manifolds.

The three-class approach is used for progressive disorders when clinicians and researchers want to diagnose or classify subjects as members of one of three ordered categories based on a continuous diagnostic marker. The decision thresholds or optimal cut-off points required for this classification are often chosen to maximize the generalized Youden index (Nakas et al., Stat Med 2013; 32: 995-1003). The effectiveness of these chosen cut-off points can be evaluated by estimating their corresponding true class fractions and their associated confidence regions. Recently, in the two-class case, parametric and non-parametric methods were investigated for the construction of confidence regions for the pair of the Youden-index-based optimal sensitivity and specificity fractions that can take into account the correlation introduced between sensitivity and specificity when the optimal cut-off point is estimated from the data (Bantis et al., Biomet 2014; 70: 212-223). A parametric approach based on the Box-Cox transformation to normality often works well while for markers having more complex distributions a non-parametric procedure using logspline density estimation can be used instead. The true class fractions that correspond to the optimal cut-off points estimated by the generalized Youden index are correlated similarly to the two-class case. In this article, we generalize these methods to the three- and to the general k-class case which involves the classification of subjects into three or more ordered categories, where ROC surface or ROC manifold methodology, respectively, is typically employed for the evaluation of the discriminatory capacity of a diagnostic marker. We obtain three- and multi-dimensional joint confidence regions for the optimal true class fractions. We illustrate this with an application to the Trail Making Test Part A that has been used to characterize cognitive impairment in patients with Parkinson's disease.

Construction of confidence regions in the ROC space after the estimation of the optimal Youden index-based cut-off point.

After establishing the utility of a continuous diagnostic marker investigators will typically address the question of determining a cut-off point which will be used for diagnostic purposes in clinical decision making. The most commonly used optimality criterion for cut-off point selection in the context of ROC curve analysis is the maximum of the Youden index. The pair of sensitivity and specificity proportions that correspond to the Youden index-based cut-off point characterize the performance of the diagnostic marker. Confidence intervals for sensitivity and specificity are routinely estimated based on the assumption that sensitivity and specificity are independent binomial proportions as they arise from the independent populations of diseased and healthy subjects, respectively. The Youden index-based cut-off point is estimated from the data and as such the resulting sensitivity and specificity proportions are in fact correlated. This correlation needs to be taken into account in order to calculate confidence intervals that result in the anticipated coverage. In this article we study parametric and non-parametric approaches for the construction of confidence intervals for the pair of sensitivity and specificity proportions that correspond to the Youden index-based optimal cut-off point. These approaches result in the anticipated coverage under different scenarios for the distributions of the healthy and diseased subjects. We find that a parametric approach based on a Box-Cox transformation to normality often works well. For biomarkers following more complex distributions a non-parametric procedure using logspline density estimation can be used.

Class II subdivision treatment with the Herbst appliance.

OBJECTIVE: To assess the effectiveness of Class II subdivision Herbst nonextraction treatment and its short-term stability retrospectively. MATERIALS AND METHODS: Twenty-two Class II subdivision (SUB: right-left molar difference ≥0.75 cusp width) and 22 symmetric Class II patients (SYM: ≥0.75 cusp width bilaterally) were matched according to gender and pretreatment handwrist radiographic stage. The mean treatment duration of the Herbst and subsequent multibracket phase was 8 months and 14 months, respectively. The mean retention period amounted to 36 months. Dental casts from before treatment (T1), after Herbst treatment (T2), after Multibracket treatment (T3), and after retention (T4) were evaluated. RESULTS: A bilateral Class I or super Class I molar relationship was seen in 72.7% (SUB) and 77.3% (SYM) at T3. The corresponding values at T4 were 63.7% (SUB) and 72.7% (SYM). A unilateral or bilateral Class III molar relationship was more frequent in the SUB group (T3: +4.6%; T4: +13.6%). For overjet, similar mean values were seen in both groups after treatment (T3: SUB, 2.7 mm; SYM, 2.3 mm) and after retention (T4: SUB, 3.0 mm; SYM, 3.4 mm). This was also true for the midline shift (T3: SUB, -0.4 mm; SYM, 0.0 mm; T4: SUB, -0.3 mm; SYM, 0.0 mm). CONCLUSION: Class II subdivision Herbst treatment was successful similarly to symmetric Class II Herbst treatment. However, a slight overcompensation of the molar relationship (Class III tendency) was more frequent in the subdivision patients (original Class I side).

High prevalence of childhood asthma in Northern Israel is linked to air pollution by particulate matter: evidence from GIS analysis and Bayesian Model Averaging.

The medical records of 3922 school children residing in the Greater Haifa Metropolitan Area in Northern Israel were analyzed. Individual exposure to ambient air pollution (SO(2) and PM(10)) for each child was estimated using Geographic Information Systems tools. Factors affecting childhood asthma risk were then investigated using logistic regression and the more recently developed Bayesian Model Averaging (BMA) tools. The analysis reveals that childhood asthma in the study area appears to be significantly associated with particulate matter of less than 10 µm in aerodynamic diameter (PM(10)) (Odds Ratio (OR) = .11; P<0.001). However, no significant association with asthma prevalence was found for SO(2) (P >0.2), when PM(10) and SO(2) were introduced into the models simultaneously. When considering a change in PM(10) between the least and the most polluted parts of the study area (9.4 µg/m(3)), the corresponding OR, calculated using the BMA analysis, is 2.58 (with 95% posterior probability limits of OR ranging from 1.52 to 4.41), controlled for gender, age, proximity to main roads, the town of a child's residence, and family's socio-economic status. Thus, it is concluded that exposure to airborne particular matter, even at relatively low concentrations (40-50 µg/m(3)), generally below international air pollution standards (55-70 µg/m(3)), appears to be a considerable risk factor for childhood asthma in urban areas. This should be a cause of concern for public health authorities and environmental decision-makers.

Estimation of the ROC curve under verification bias.

The ROC (receiver operating characteristic) curve is the most commonly used statistical tool for describing the discriminatory accuracy of a diagnostic test. Classical estimation of the ROC curve relies on data from a simple random sample from the target population. In practice, estimation is often complicated due to not all subjects undergoing a definitive assessment of disease status (verification). Estimation of the ROC curve based on data only from subjects with verified disease status may be badly biased. In this work we investigate the properties of the doubly robust (DR) method for estimating the ROC curve under verification bias originally developed by Rotnitzky, Faraggi and Schisterman (2006) for estimating the area under the ROC curve. The DR method can be applied for continuous scaled tests and allows for a non-ignorable process of selection to verification. We develop the estimator's asymptotic distribution and examine its finite sample properties via a simulation study. We exemplify the DR procedure for estimation of ROC curves with data collected on patients undergoing electron beam computer tomography, a diagnostic test for calcification of the arteries.

Youden Index and the optimal threshold for markers with mass at zero.

The Youden Index is often used as a summary measure of the receiver operating characteristic curve. It measures the effectiveness of a diagnostic marker and permits the selection of an optimal threshold value or cutoff point for the biomarker of interest. Some markers, while basically continuous and positive, have a spike or positive mass of probability at the value zero. We provide a flexible modeling approach for estimating the Youden Index and its associated cutoff point for such spiked data and compare it with the standard empirical approach. We show how this modeling approach can be adjusted to take covariate information into account. This approach is applied to data on the Coronary Calcium Score, a marker for atherosclerosis.

Comparing the areas under two correlated ROC curves: parametric and non-parametric approaches.

In order to compare the discriminatory effectiveness of two diagnostic markers the equality of the areas under the respective Receiver Operating Characteristic Curves is commonly tested. A non-parametric test based on the Mann-Whitney statistic is generally used. Weiand et al. (1989) present a parametric test based on normal distributional assumptions. We extend this test using the Box-Cox power family of transformations to non-normal situations. These three test procedures are compared in terms of significance level and power by means of a large simulation study. Overall we find that transforming to normality is to be preferred. An example of two pancreatic cancer serum biomarkers is used to illustrate the methodology.

Inference for the dependent competing risks model with masked causes of failure.

The competing risks model is useful in settings in which individuals/units may die/fail for different reasons. The cause specific hazard rates are taken to be piecewise constant functions. A complication arises when some of the failures are masked within a group of possible causes. Traditionally, statistical inference is performed under the assumption that the failure causes act independently on each item. In this paper we propose an EM-based approach which allows for dependent competing risks and produces estimators for the sub-distribution functions. We also discuss identifiability of parameters if none of the masked items have their cause of failure clarified in a second stage analysis (e.g. autopsy). The procedures proposed are illustrated with two datasets.

ROC analysis for markers with mass at zero.

The receiver operating characteristic (ROC) curve and in particular the area under the curve (AUC) is commonly used to examine the discriminatory ability of diagnostic markers. Certain markers while basically continuous and non-negative have a positive probability mass (spike) at the value zero. We discuss a flexible modelling approach to such data and contrast it with the standard non-parametric approach. We show how the modelling approach can be extended to take account of the effect of explanatory variables. We motivate this problem and illustrate the modelling approach using data on the coronary calcium score, measured by electron beam tomography, which is a marker for atherosclerosis.

Estimation of the Youden Index and its associated cutoff point.

The Youden Index is a frequently used summary measure of the ROC (Receiver Operating Characteristic) curve. It both, measures the effectiveness of a diagnostic marker and enables the selection of an optimal threshold value (cutoff point) for the marker. In this paper we compare several estimation procedures for the Youden Index and its associated cutoff point. These are based on (1) normal assumptions; (2) transformations to normality; (3) the empirical distribution function; (4) kernel smoothing. These are compared in terms of bias and root mean square error in a large variety of scenarios by means of an extensive simulation study. We find that the empirical method which is the most commonly used has the overall worst performance. In the estimation of the Youden Index the kernel is generally the best unless the data can be well transformed to achieve normality whereas in estimation of the optimal threshold value results are more variable.

Adjusting the generalized ROC curve for covariates.

Receiver operating characteristic (ROC) curves and in particular the area under the curve (AUC), are widely used to examine the effectiveness of diagnostic markers. Diagnostic markers and their corresponding ROC curves can be strongly influenced by covariate variables. When several diagnostic markers are available, they can be combined by a best linear combination such that the area under the ROC curve of the combination is maximized among all possible linear combinations. In this paper we discuss covariate effects on this linear combination assuming that the multiple markers, possibly transformed, follow a multivariate normal distribution. The ROC curve of this linear combination when markers are adjusted for covariates is estimated and approximate confidence intervals for the corresponding AUC are derived. An example of two biomarkers of coronary heart disease for which covariate information on age and gender is available is used to illustrate this methodology.

ROC curve analysis for biomarkers based on pooled assessments.

Interleukin-6 is a biomarker of inflammation which has been suggested as having potential discriminatory ability for myocardial infarction. Because of its high assaying cost it is very expensive to evaluate this marker. In order to reduce this cost we propose pooling the specimens. In this paper we examine the efficiency of ROC curve analysis, specifically the estimation of the area under the ROC curve, when dealing with pooled data. We study the effect of pooling when there are only a fixed number of individuals available for testing and pooling is carried out to save on the number of assays. Alternatively we examine how many pooled assays of size g are necessary to provide essentially the same information as N individual assays. We measure loss of information by means of the change in root mean square error of the estimate of the area under the ROC curve and study the extent of this loss via a simulation study.

Estimation of the area under the ROC curve.

The area under the receiver operating characteristic curve is frequently used as a measure for the effectiveness of diagnostic markers. In this paper we discuss and compare estimation procedures for this area. These are based on (i) the Mann-Whitney statistic; (ii) kernel smoothing; (iii) normal assumptions; (iv) empirical transformations to normality. These are compared in terms of bias and root mean square error in a large variety of situations by means of an extensive simulation study. Overall we find that transforming to normality usually is to be preferred except for bimodal cases where kernel methods can be effective.

Parametric modeling for survival with competing risks and masked failure causes.

We consider a life testing situation in which systems are subject to failure from independent competing risks. Following a failure, immediate (stage-1) procedures are used in an attempt to reach a definitive diagnosis. If these procedures fail to result in a diagnosis, this phenomenon is called masking. Stage-2 procedures, such as failure analysis or autopsy, provide definitive diagnosis for a sample of the masked cases. We show how stage-1 and stage-2 information can be combined to provide statistical inference about (a) survival functions of the individual risks, (b) the proportions of failures associated with individual risks and (c) probability, for a specified masked case, that each of the masked competing risks is responsible for the failure. Our development is based on parametric distributional assumptions and the special case for which the failure times for the competing risks have a Weibull distribution is discussed in detail.