DxGoals: A Software Tool for Determining and Analyzing Clinically Meaningful Classification Accuracy Goals for Diagnostic Tests.

BACKGROUND: To evaluate diagnostic tests for low prevalence conditions, classification accuracy metrics such as sensitivity, specificity, and positive likelihood ratio (PLR) and negative likelihood ratio (NLR) are advantageous because they are prevalence-independent and thus estimable in studies enriched for the condition. However, classification accuracy goals are often chosen without a clear understanding of whether they are clinically meaningful. Pennello (2021) proposed a risk stratification framework for determining classification accuracy goals. A software application is needed to determine the goals and provide data analysis. METHODS: We introduce DxGoals, a freely available, R-Shiny software application for determining, visualizing, and analyzing classification accuracy goals for diagnostic tests. Given prevalence p for the target condition and specification that a test's positive and negative predictive values PPVand NPV=1-cNPV should satisfy PPV>PPV* and cNPV

Multiparametric Quantitative Imaging Biomarkers for Phenotype Classification: A Framework for Development and Validation.

This manuscript is the third in a five-part series related to statistical assessment methodology for technical performance of multi-parametric quantitative imaging biomarkers (mp-QIBs). We outline approaches and statistical methodologies for developing and evaluating a phenotype classification model from a set of multiparametric QIBs. We then describe validation studies of the classifier for precision, diagnostic accuracy, and interchangeability with a comparator classifier. We follow with an end-to-end real-world example of development and validation of a classifier for atherosclerotic plaque phenotypes. We consider diagnostic accuracy and interchangeability to be clinically meaningful claims for a phenotype classification model informed by mp-QIB inputs, aiming to provide tools to demonstrate agreement between imaging-derived characteristics and clinically established phenotypes. Understanding that we are working in an evolving field, we close our manuscript with an acknowledgement of existing challenges and a discussion of where additional work is needed. In particular, we discuss the challenges involved with technical performance and analytical validation of mp-QIBs. We intend for this manuscript to further advance the robust and promising science of multiparametric biomarker development.

Multiparametric Quantitative Imaging Biomarker as a Multivariate Descriptor of Health: A Roadmap.

Multiparametric quantitative imaging biomarkers (QIBs) offer distinct advantages over single, univariate descriptors because they provide a more complete measure of complex, multidimensional biological systems. In disease, where structural and functional disturbances occur across a multitude of subsystems, multivariate QIBs are needed to measure the extent of system malfunction. This paper, the first Use Case in a series of articles on multiparameter imaging biomarkers, considers multiple QIBs as a multidimensional vector to represent all relevant disease constructs more completely. The approach proposed offers several advantages over QIBs as multiple endpoints and avoids combining them into a single composite that obscures the medical meaning of the individual measurements. We focus on establishing statistically rigorous methods to create a single, simultaneous measure from multiple QIBs that preserves the sensitivity of each univariate QIB while incorporating the correlation among QIBs. Details are provided for metrological methods to quantify the technical performance. Methods to reduce the set of QIBs, test the superiority of the mp-QIB model to any univariate QIB model, and design study strategies for generating precision and validity claims are also provided. QIBs of Alzheimer's Disease from the ADNI merge data set are used as a case study to illustrate the methods described.

Development and evaluation of safety and effectiveness of novel cancer screening tests for routine clinical use with applications to multicancer detection technologies.

Multicancer screening is a promising approach to improving the detection of preclinical disease, but current technologies have limited ability to identify precursor or early stage lesions, and approaches for developing the evidentiary chain are unclear. Frameworks to enable development and evaluation from discovery through evidence of clinical effectiveness are discussed.

Analytical and clinical evaluation of biomarkers assays: when are biomarkers ready for prime time?

BACKGROUND: Biomarker assays can be evaluated for analytical performance (ability of assay to measure the biomarker quantity) and clinical performance (ability of assay result to inform of the clinical condition of interest). Additionally, a biomarker assay is said to have clinical utility if it ultimately improves patient outcomes when used as intended. PURPOSE: This article reviews analytical and clinical performance studies of biomarker assay tests and also some designs of clinical utility studies. RESULTS: Appropriate design and statistical analysis of analytical and clinical evaluation studies depend on the intended clinical use of the test. Some key aspects to valid performance studies include using subjects who are independent of those used to develop the test, masking users of the test to any other available test or reference results, and including in the primary statistical analysis subjects with unavailable results in an intention-to-diagnose analysis. Ingenuity in study design and analysis may be required for efficient and unbiased estimation of performance. LIMITATIONS: Performance studies need to be carefully planned as they can be prone to many sources of bias. Analytical inaccuracy can hamper the clinical performance of biomarkers. CONCLUSIONS: As biomedical research and technology advance, challenges in study design and statistical analysis will continue to emerge for analytical and clinical performance studies of biomarker assays. Although not emphasized in some circles, the analytical performance of a biomarker assay is important to characterize. Analytical performance studies have many study design and statistical analysis challenges that deserve further attention.

Bayesian analysis of diagnostic test accuracy when disease state is unverified for some subjects.

Studies of the accuracy of medical tests to diagnose the presence or absence of disease can suffer from an inability to verify the true disease state in everyone. When verification is missing at random (MAR), the missing data mechanism can be ignored in likelihood-based inference. However, this assumption may not hold even approximately. When verification is nonignorably missing, the most general model of the distribution of disease state, test result, and verification indicator is overparameterized. Parameters are only partially identified, creating regions of ignorance for maximum likelihood estimators. For studies of a single test, we use Bayesian analysis to implement the most general nonignorable model, a reduced nonignorable model with identifiable parameters, and the MAR model. Simple Gibbs sampling algorithms are derived that enable computation of the posterior distribution of test accuracy parameters. In particular, the posterior distribution is easily obtained for the most general nonignorable model, which makes relatively weak assumptions about the missing data mechanism. For this model, the posterior distribution combines two sources of uncertainty: ignorance in the estimation of partially identified parameters, and imprecision due to finite sampling variability. We compare the three models on data from a study of the accuracy of scintigraphy to diagnose liver disease.

Multireader multicase variance analysis for binary data.

Multireader multicase (MRMC) variance analysis has become widely utilized to analyze observer studies for which the summary measure is the area under the receiver operating characteristic (ROC) curve. We extend MRMC variance analysis to binary data and also to generic study designs in which every reader may not interpret every case. A subset of the fundamental moments central to MRMC variance analysis of the area under the ROC curve (AUC) is found to be required. Through multiple simulation configurations, we compare our unbiased variance estimates to naïve estimates across a range of study designs, average percent correct, and numbers of readers and cases.

Duncan's k-ratio Bayes rule approach to multiple comparisons: an overview.

An alternative to frequentist approaches to multiple comparisons is Duncan's k-ratio Bayes rule approach. The purpose of this paper is to compile key results on k-ratio Bayes rules for a number of multiple comparison problems that heretofore, have only been available in separate papers or doctoral dissertations. Among other problems, multiple comparisons for means in one-way, two-way, and treatments-vs.-control structures will be reviewed. In the k-ratio approach, the optimal joint rule for a multiple comparisons problem is derived under the assumptions of additive losses and prior exchangeability for the component comparisons. In the component loss function for a comparison, a balance is achieved between the decision losses due to Type I and Type II errors by assuming that their ratio is k. The component loss is also linear in the magnitude of the error. Under the assumption of additive losses, the joint Bayes rule for the component comparisons applies to each comparison the Bayes test for that comparison considered alone. That is, a comparisonwise approach is optimal. However, under prior exchangeability of the comparisons, the component test critical regions adapt to omnibus patterns in the data. For example, for a balanced one-way array of normally distributed means, the Bayes critical t value for a difference between means is inversely related to the F ratio measuring heterogeneity among the means, resembling a continuous version of Fisher's F-protected least significant difference rule. For more complicated treatment structures, the Bayes critical t value for a difference depends intuitively on multiple F ratios and marginal difference(s) (if applicable), such that the critical t value warranted for the difference can range from being as conservative as that given by a familywise rule to actually being anti-conservative relative to that given by the unadjusted 5%-level Student's t test.