Face identification proficiency test designed using item response theory.

Measures of face-identification proficiency are essential to ensure accurate and consistent performance by professional forensic face examiners and others who perform face-identification tasks in applied scenarios. Current proficiency tests rely on static sets of stimulus items and so cannot be administered validly to the same individual multiple times. To create a proficiency test, a large number of items of "known" difficulty must be assembled. Multiple tests of equal difficulty can be constructed then using subsets of items. We introduce the Triad Identity Matching (TIM) test and evaluate it using item response theory (IRT). Participants view face-image "triads" (N = 225) (two images of one identity, one image of a different identity) and select the different identity. In Experiment 3, university students (N = 197) showed wide-ranging accuracy on the TIM test, and IRT modeling demonstrated that the TIM items span various difficulty levels. In Experiment 3, we used IRT-based item metrics to partition the test into subsets of specific difficulties. Simulations showed that subsets of the TIM items yielded reliable estimates of subject ability. In Experiments 3a and b, we found that the student-derived IRT model reliably evaluated the ability of non-student participants and that ability generalized across different test sessions. In Experiment 3c, we show that TIM test performance correlates with other common face-recognition tests. In summary, the TIM test provides a starting point for developing a framework that is flexible and calibrated to measure proficiency across various ability levels (e.g., professionals or populations with face-processing deficits).

A ROC-based test for evaluating the group difference with an application to neonatal audiology screening.

This article proposes a powerful method to compare two samples. The proposed method handles comparison of data by drawing inference from ROC curve model parameters. The method estimates parameters from a linear model framework on the empirical sensitivities and specificities. The consistent ROC parameters are then used to give a more powerful test than existing methods in several situations. In addition, we present a comprehensive statistic based on the Cauchy combination, which works well in all scenarios considered in this article. We also offer an efficient one-layer wild permutation procedure to calculate the P-value of our statistic. The method is particularly useful when the underlying continuous biomarker results are non-normal. We illustrate the proposed methods in a neonatal audiology diagnostic example.

Order-restricted inference for clustered ROC data with application to fingerprint matching accuracy.

Receiver operating characteristic (ROC) curve is commonly used to evaluate and compare the accuracy of classification methods or markers. Estimating ROC curves has been an important problem in various fields including biometric recognition and diagnostic medicine. In real applications, classification markers are often developed under two or more ordered conditions, such that a natural stochastic ordering exists among the observations. Incorporating such a stochastic ordering into estimation can improve statistical efficiency (Davidov and Herman, 2012). In addition, clustered and correlated data arise when multiple measurements are gleaned from the same subject, making estimation of ROC curves complicated due to within-cluster correlations. In this article, we propose to model the ROC curve using a weighted empirical process to jointly account for the order constraint and within-cluster correlation structure. The algebraic properties of resulting summary statistics of the ROC curve such as its area and partial area are also studied. The algebraic expressions reduce to the ones by Davidov and Herman (2012) for independent observations. We derive asymptotic properties of the proposed order-restricted estimators and show that they have smaller mean-squared errors than the existing estimators. Simulation studies also demonstrate better performance of the newly proposed estimators over existing methods for finite samples. The proposed method is further exemplified with the fingerprint matching data from the National Institute of Standards and Technology Special Database 4.

Group sequential comparison of positive predictive value curves for correlated biomarker data.

Clinical studies of predictive diagnostic tests consider the evaluation of a single test and comparison of two tests regarding their predictive accuracy of disease status. The positive predictive value (PPV) curve is used for assessing the probability of predicting the disease given a positive test result. The sequential property of one PPV curve had been studied. However, in later stages of diagnostic test development, it is more interesting to compare predictive accuracy of two tests. In this article, we propose a group sequential test for the comparison of PPV curves for paired designs when both diagnostic tests are applied to the same subject. We first derive asymptotic properties of the sequential differences of two correlated empirical PPV curves under the common case-control sampling. We then apply these results to develop a group sequential test procedure. The asymptotic results are also critical for deriving both the optimal sample size ratio and minimal required sample sizes for the proposed procedure. Our simulation studies show that the proposed sequential testing maintains the nominal type I error rate in finite samples. The proposed design is illustrated in a hypothetical lung cancer predictive trial and in a cancer diagnostic trial.

A cluster-adjusted rank-based test for a clinical trial concerning multiple endpoints with application to dietary intervention assessment.

Multiple endpoints are often naturally clustered based on their scientific interpretations. Tests that compare these clustered outcomes between independent groups may lose efficiency if the cluster structures are not properly accounted for. For the two-sample generalized Behrens-Fisher hypothesis concerning multiple endpoints we propose a cluster-adjusted multivariate test procedure for the comparison and demonstrate its gain in efficiency over test procedures that ignore the clusters. Data from a dietary intervention trial are used to illustrate the methods.

Evaluating accuracy of diagnostic tests without conditional independence assumption.

Evaluating the accuracy (ie, estimating the sensitivity and specificity) of new diagnostic tests without the presence of a gold standard is of practical meaning and has been the subject of intensive study for several decades. Existing methods use 2 or more diagnostic tests under several basic assumptions and then estimate the accuracy parameters via the maximum likelihood estimation. One of the basic assumptions is the conditional independence of the tests given the disease status. This assumption is impractical in many real applications in veterinary research. Several methods have been proposed with various dependence models to relax this assumption. However, these methods impose subjective dependence structures, which may not be practical and may introduce additional nuisance parameters. In this article, we propose a simple method for addressing this problem without the conditional independence assumption, using an empirical conditioning approach. The proposed method reduces to the popular Hui-Walter model in the case of conditional independence. Also, our likelihood function is of order-2 polynomial in parameters, while that of Hui-Walter is of order-3. The reduced model complexity increases the stability in estimation. Simulation studies are conducted to evaluate the performance of the proposed method, which shows overall smaller biases in estimation and is more stable than the existing method, especially when tests are conditionally dependent. Two real data examples are used to illustrate the proposed method.

GATE: an efficient procedure in study of pleiotropic genetic associations.

BACKGROUND: The association studies on human complex traits are admittedly propitious to identify deleterious genetic markers. Compared to single-trait analyses, multiple-trait analyses can arguably make better use of the information on both traits and markers, and thus improve statistical power of association tests prominently. Principal component analysis (PCA) is a well-known useful tool in multivariate analysis and can be applied to this task. Generally, PCA is first performed on all traits and then a certain number of top principal components (PCs) that explain most of the trait variations are selected to construct the test statistics. However, under some situations, only utilizing these top PCs would lead to a loss of important evidences from discarded PCs and thus makes the capability compromised. METHODS: To overcome this drawback while keeping the advantages of using the top PCs, we propose a group accumulated test evidence (GATE) procedure. By dividing the PCs which is sorted in the descending order according to the corresponding eigenvalues into a few groups, GATE integrates the information of traits at the group level. RESULTS: Simulation studies demonstrate the superiority of the proposed approach over several existing methods in terms of statistical power. Sometimes, the increase of power can reach 25%. These methods are further illustrated using the Heterogeneous Stock Mice data which is collected from a quantitative genome-wide association study. CONCLUSIONS: Overall, GATE provides a powerful test for pleiotropic genetic associations.

Regularized Bayesian calibration and scoring of the WD-FAB IRT model improves predictive performance over marginal maximum likelihood.

Item response theory (IRT) is the statistical paradigm underlying a dominant family of generative probabilistic models for test responses, used to quantify traits in individuals relative to target populations. The graded response model (GRM) is a particular IRT model that is used for ordered polytomous test responses. Both the development and the application of the GRM and other IRT models require statistical decisions. For formulating these models (calibration), one needs to decide on methodologies for item selection, inference, and regularization. For applying these models (test scoring), one needs to make similar decisions, often prioritizing computational tractability and/or interpretability. In many applications, such as in the Work Disability Functional Assessment Battery (WD-FAB), tractability implies approximating an individual's score distribution using estimates of mean and variance, and obtaining that score conditional on only point estimates of the calibrated model. In this manuscript, we evaluate the calibration and scoring of models under this common use-case using Bayesian cross-validation. Applied to the WD-FAB responses collected for the National Institutes of Health, we assess the predictive power of implementations of the GRM based on their ability to yield, on validation sets of respondents, ability estimates that are most predictive of patterns of item responses. Our main finding indicates that regularized Bayesian calibration of the GRM outperforms the regularization-free empirical Bayesian procedure of marginal maximum likelihood. We also motivate the use of compactly supported priors in test scoring.

A resample-replace lasso procedure for combining high-dimensional markers with limit of detection.

In disease screening, a biomarker combination developed by combining multiple markers tends to have a higher sensitivity than an individual marker. Parametric methods for marker combination rely on the inverse of covariance matrices, which is often a non-trivial problem for high-dimensional data generated by modern high-throughput technologies. Additionally, another common problem in disease diagnosis is the existence of limit of detection (LOD) for an instrument - that is, when a biomarker's value falls below the limit, it cannot be observed and is assigned an NA value. To handle these two challenges in combining high-dimensional biomarkers with the presence of LOD, we propose a resample-replace lasso procedure. We first impute the values below LOD and then use the graphical lasso method to estimate the means and precision matrices for the high-dimensional biomarkers. The simulation results show that our method outperforms alternative methods such as either substitute NA values with LOD values or remove observations that have NA values. A real case analysis on a protein profiling study of glioblastoma patients on their survival status indicates that the biomarker combination obtained through the proposed method is more accurate in distinguishing between two groups.

Tire Classification by Elemental Signatures Using Laser-Induced Breakdown Spectroscopy.

Tire evidence is a form of trace evidence that is often overlooked in today's forensics, while frequently found at crime or accident scenes, usually in the form of skid marks. The pattern of the tire skid mark has been used before to link a tire or car to a scene, but the widespread use of anti-lock braking systems makes this an almost impossible and abandoned route of analysis. With this in mind, using the chemical profile of a tire has potential to link a car or tire back to a scene in which its trace material is found. This study shows the successful use of the elemental profile of tire rubber to classify 32 different samples using laser-induced breakdown spectroscopy, analyzed by principal component analysis combined with linear discriminant analysis. A classification accuracy close to 99% shows the ever-growing use of laser-induced breakdown spectroscopy as a technique of choice for forensic analysis of tire rubber, opening the path for its use as a forensic evidence.

Validating Nonlinear Registration to Improve Subtraction Images for Lesion Detection and Quantification in Multiple Sclerosis.

BACKGROUND AND PURPOSE: To propose and validate nonlinear registration techniques for generating subtraction images because of their ability to reduce artifacts and improve lesion detection and lesion volume quantification. METHODS: Postcontrast T1 -weighted spin echo and T2 -weighted dual echo images were acquired for 20 patients with relapsing-remitting multiple sclerosis (RRMS) on a monthly basis for a year (14 women, average age 33.6 ± 6.9). The T2 -weighted images from the first scan were used as a baseline for each patient. The images from the last scan were registered to the baseline image. Four different registration algorithms used for evaluation included; linear, halfway linear, nonlinear, and nonlinear halfway. Subtraction images were generated after brain extraction, intensity normalization, and Gaussian blurring. Lesion activity changes along with identified artifacts were scored on all four techniques by two independent observers. Additionally, quantitative analysis of the algorithms was performed by estimating the volume changes of simulated lesions and real lesions. For real lesion volume change analysis, five subjects were selected randomly. Subtraction images were generated between all the 11 time points and the baseline image using linear and nonlinear registration for the five subjects. RESULTS: Lesion activity detection resulted in similar performance among the four registration techniques. Lesion volume measurements on subtraction images using nonlinear registration were closer to lesion volume on T2 -weighted images. A statistically significant difference was observed among the four registration techniques while evaluating yin-yang artifacts. Pairwise comparisons showed that nonlinear registration results in the least amount of yin-yang artifacts, which are significantly different. CONCLUSIONS: Nonlinear registration for generation of subtraction images has been demonstrated to be a promising new technique as it shows improvement in lesion activity change detection. This approach decreases the number of artifacts in subtraction images. With improved lesion volume estimates and reduced artifacts, nonlinear registration may lead to discarding less subject data and an improvement in the statistical power of subtraction imaging studies.