Sample size calculation for comparing two screening tests when the gold standard is missing at random.

Sample size formulas have been proposed for comparing two sensitivities (specificities) in the presence of verification bias under a paired design. However, the existing sample size formulas involve lengthy calculations of derivatives and are too complicated to implement. In this paper, we propose alternative sample size formulas for each of three existing tests, two Wald tests and one weighted McNemar's test. The proposed sample size formulas are more intuitive and simpler to implement than their existing counterparts. Furthermore, by comparing the sample sizes calculated based on the three tests, we can show that the three tests have similar sample sizes even though the weighted McNemar's test only use the data from discordant pairs whereas the two Wald tests also use the additional data from accordant pairs.

Direct estimation of volume under the ROC surface with verification bias.

In practice, the receiver operating characteristic (ROC) curve of a diagnostic test is widely used to show the performance of the test for discriminating two-class events. The area under the ROC curve (AUC) is proposed as an index for the assessment of the diagnostic accuracy of the test under consideration. Due to ethical and cost considerations associated with application of gold standard (GS) tests, only a subset of the patients initially tested have verified disease status. Statistical evaluation of the test performance based only on test results from subjects with verified disease status are typically biased. Various AUC estimation methods for tests with verification biased data have been developed over the last few decades. In this article, we develop new direct estimation methods for the volume under the ROC surface (VUS) by extending the AUC estimation methods for two-class diagnostic tests to three-class diagnostic tests in the presence of verification bias. The proposed methods will provide a comprehensive guide to deal with the verification bias in three-class diagnostic test accuracy studies and lead to a better choice of diagnostic tests.

Verification Bias Correction in Endometrial Abnormalities in Infertile Women Referred to Royan Institute Using Statistical Methods.

Background: Verification bias is a common bias in the diagnostic accuracy of diagnostic tests and occurs when a number of individuals do not perform the gold standard test. In this study, we review the correcting methods of verification bias. Methods: In a cross-sectional study in 2020, 567 infertile women who were referred to Royan Research Institute were evaluated. The ultrasound is the performed test and the gold standard are hysteroscopy for some, and pathology for other abnormalities. For correcting verification bias conventional, Begg and Greens, Zhou, and logistic regression methods were used. Results: In the gold standard hysteroscopy test, the sensitivity (SEN) and specificity (SPEC) obtained in conventional, Begg and Greens, Zhou, and logistics Regression methods were (50%, 90.3%), (48%, 96%), (22%, 77%), (50%, 90%), and (72.8, 77) respectively. Furthermore, the area under the curve (AUC) index and kappa statistics were calculated as 70.2%, and 43.6% respectively. In the pathology gold standard test, the SEN and SPEC for the conventional methods, Begg and Greens, Zhou and logistics regression were (67.7%, 86.7%), (66%, 88%), (29%, 70%), (66.9%, 87.6%), and (73%, 83.9%) respectively. Also, the AUC index and kappa statistics were 77%, and 55% respectively. Conclusion: In the study on endometrial abnormalities in infertile women, assuming that the missing data mechanism is random, the amount of bias in calculating SEN and SPEC is very low in the diagnostic tests calculated before and after correction, using Begg and Greens and logistic regression method. But Zhou's method gives rather large biased estimates.

Weighted McNemar's test for the comparison of two screening tests in the presence of verification bias.

Statistical methods have been well-developed for comparing two binary screening tests in the presence of verification bias. However, the complexity of existing methods and the computational difficulty in implementing them have restricted their use. A simple and easily implemented statistical method is therefore needed. In this paper, we propose a weighted McNemar's test statistic for comparing two sensitivities(specificities). The proposed test statistics are intuitive and simple to compute, only involving some minor modification of a McNemar's test statistic using the estimated verification probabilities for discordant pairs. Simulations demonstrate that the proposed weighted McNemar's test statistics preserve type I error as well as or better than the existing statistics. Furthermore, unlike the existing methods, the proposed weighted McNemar's test statistics can still be applied even when none of the accordant pairs are verified.

Correcting for partial verification bias in diagnostic accuracy studies: A tutorial using R.

Diagnostic tests play a crucial role in medical care. Thus any new diagnostic tests must undergo a thorough evaluation. New diagnostic tests are evaluated in comparison with the respective gold standard tests. The performance of binary diagnostic tests is quantified by accuracy measures, with sensitivity and specificity being the most important measures. In any diagnostic accuracy study, the estimates of these measures are often biased owing to selective verification of the patients, which is referred to as partial verification bias. Several methods for correcting partial verification bias are available depending on the scale of the index test, target outcome, and missing data mechanism. However, these are not easily accessible to the researchers due to the complexity of the methods. This article aims to provide a brief overview of the methods available to correct for partial verification bias involving a binary diagnostic test and provide a practical tutorial on how to implement the methods using the statistical programming language R.

Weighted generalized score test for comparing predictive values in the presence of verification bias.

Positive and negative predictive values of a diagnostic test are two important measures of test accuracy, which are more relevant in clinical settings than sensitivity and specificity. Statistical methods have been well-developed to compare the predictive values of two binary diagnostic tests when test results and disease status fully observed for all study patients. In practice, however, it is common that only a subset of study patients have the disease status verified due to ethical or cost considerations. Methods applied directly to the verified subjects may lead to biased results. A bias-corrected method has been developed to compare two predictive values in the presence of verification bias. However, the complexity of the existing method and the computational difficulty in implementing it has restricted its use. A simple and easily implemented statistical method is therefore needed. In this paper, we propose a weighted generalized score (WGS) test statistic for comparing two predictive values in the presence of verification bias. The proposed WGS test statistic is intuitive and simple to compute, only involving some minor modification of the WGS test statistic when disease status is verified for each study patient. Simulations demonstrate that the proposed WGS test statistic preserves type I error much better than the existing Wald statistic. The method is illustrated with data from a study of methods for the diagnosis of coronary artery disease.

Adjusting for verification bias in diagnostic accuracy measures when comparing multiple screening tests - an application to the IP1-PROSTAGRAM study.

INTRODUCTION: Novel screening tests used to detect a target condition are compared against either a reference standard or other existing screening methods. However, as it is not always possible to apply the reference standard on the whole population under study, verification bias is introduced. Statistical methods exist to adjust estimates to account for this bias. We extend common methods to adjust for verification bias when multiple tests are compared to a reference standard using data from a prospective double blind screening study for prostate cancer. METHODS: Begg and Greenes method and multiple imputation are extended to include the results of multiple screening tests which determine condition verification status. These two methods are compared to the complete case analysis using the IP1-PROSTAGRAM study data. IP1-PROSTAGRAM used a paired-cohort double-blind design to evaluate the use of imaging as alternative tests to screen for prostate cancer, compared to a blood test called prostate specific antigen (PSA). Participants with positive imaging (index) and/or PSA (control) underwent a prostate biopsy (reference standard). RESULTS: When comparing complete case results to Begg and Greenes and methods of multiple imputation there is a statistically significant increase in the specificity estimates for all screening tests. Sensitivity estimates remained similar across the methods, with completely overlapping 95% confidence intervals. Negative predictive value (NPV) estimates were higher when adjusting for verification bias, compared to complete case analysis, even though the 95% confidence intervals overlap. Positive predictive value (PPV) estimates were similar across all methods. CONCLUSION: Statistical methods are required to adjust for verification bias in accuracy estimates of screening tests. Expanding Begg and Greenes method to include multiple screening tests can be computationally intensive, hence multiple imputation is recommended, especially as it can be modified for low prevalence of the target condition.

Statistical methods for evaluating the fine needle aspiration cytology procedure in breast cancer diagnosis.

BACKGROUND: Statistical issues present while evaluating a diagnostic procedure for breast cancer are non rare but often ignored, leading to biased results. We aimed to evaluate the diagnostic accuracy of the fine needle aspiration cytology(FNAC), a minimally invasive and rapid technique potentially used as a rule-in or rule-out test, handling its statistical issues: suspect test results and verification bias. METHODS: We applied different statistical methods to handle suspect results by defining conditional estimates. When considering a partial verification bias, Begg and Greenes method and multivariate imputation by chained equations were applied, however, and a Bayesian approach with respect to each gold standard was used when considering a differential verification bias. At last, we extended the Begg and Greenes method to be applied conditionally on the suspect results. RESULTS: The specificity of the FNAC test above 94%, was always higher than its sensitivity regardless of the proposed method. All positive likelihood ratios were higher than 10, with variations among methods. The positive and negative yields were high, defining precise discriminating properties of the test. CONCLUSION: The FNAC test is more likely to be used as a rule-in test for diagnosing breast cancer. Our results contributed in advancing our knowledge regarding the performance of FNAC test and the methods to be applied for its evaluation.

Nonparametric inference of the area under ROC curve under two-phase cluster sampling.

Nonparametric inference of the area under ROC curve (AUC) has been well developed either in the presence of verification bias or clustering. However, current nonparametric methods are not able to handle cases where both verification bias and clustering are present. Such a case arises when a two-phase study design is applied to a cohort of subjects (verification bias) where each subject might have multiple test results (clustering). In such cases, the inference of AUC must account for both verification bias and intra-cluster correlation. In the present paper, we propose an IPW AUC estimator that corrects for verification bias and derive a variance formula to account for intra-cluster correlations between disease status and test results. Results of a simulation study indicate that the method that assumes independence underestimates the true variance of the IPW AUC estimator in the presence of intra-cluster correlations. The proposed method, on the other hand, provides a consistent variance estimate for the IPW AUC estimator by appropriately accounting for correlations between true disease statuses and between test results.

Direct estimation of the area under the receiver operating characteristic curve with verification biased data.

In medical diagnostic studies, verification of the true disease status might be partially missing based on results of diagnostic tests and other characteristics of subjects. Because estimates of area under the ROC curve (AUC) based on partially validated subjects are usually biased, it is usually necessary to estimate AUC from a bias-corrected ROC curve. In this article, various direct estimation methods of the AUC based on hybrid imputation [full imputations and mean score imputation (MSI)], inverse probability weighting, and the semiparametric efficient (SPE) approach are proposed and compared in the presence of verification bias when the test result is continuous under the assumption that the true disease status, if missing, is missing at random. Simulation results show that the proposed estimators are accurate for the biased sampling if the disease and verification models are correctly specified. The SPE and MSI based estimators perform well even under the misspecified disease/verification models. Numerical studies are performed to compare the finite sample performance of the proposed approaches with existing methods. A real dataset of neonatal hearing screening study is analyzed.

True verification probabilities should not be used in estimating the area under receiver operating characteristic curve.

In medical research, a two-phase study is often used for the estimation of the area under the receiver operating characteristic curve (AUC) of a diagnostic test. However, such a design introduces verification bias. One of the methods to correct verification bias is inverse probability weighting (IPW). Since the probability a subject is selected into phase 2 of the study for disease verification is known, both true and estimated verification probabilities can be used to form an IPW estimator for AUC. In this article, we derive explicit variance formula for both IPW AUC estimators and show that the IPW AUC estimator using the true values of verification probabilities even when they are known are less efficient than its counterpart using the estimated values. Our simulation results show that the efficiency loss can be substantial especially when the variance of test result in disease population is small relative to its counterpart in nondiseased population.

Bayesian nonparametric estimation of ROC surface under verification bias.

The receiver operating characteristic (ROC) surface, as a generalization of the ROC curve, has been widely used to assess the accuracy of a diagnostic test for three categories. A common problem is verification bias, referring to the situation where not all subjects have their true classes verified. In this paper, we consider the problem of estimating the ROC surface under verification bias. We adopt a Bayesian nonparametric approach by directly modeling the underlying distributions of the three categories by Dirichlet process mixture priors. We propose a robust computing algorithm by only imposing a missing at random assumption for the verification process but no assumption on the distributions. The method can also accommodate covariates information in estimating the ROC surface, which can lead to a more comprehensive understanding of the diagnostic accuracy. It can be adapted and hugely simplified in the case where there is no verification bias, and very fast computation is possible through the Bayesian bootstrap process. The proposed method is compared with other commonly used methods by extensive simulations. We find that the proposed method generally outperforms other approaches. Applying the method to two real datasets, the key findings are as follows: (1) human epididymis protein 4 has a slightly better diagnosis ability compared to CA125 in discriminating healthy, early stage, and late stage patients of epithelial ovarian cancer. (2) Serum albumin has a prognostic ability in distinguishing different stages of hepatocellular carcinoma.

Testing for Verification Bias in Reported Malignancy Risks for Side-Branch Intraductal Papillary Mucinous Neoplasms: A Simulation Modeling Approach.

OBJECTIVE: The objective of our study was to test for the possibility that published malignancy risks for side-branch intraductal papillary mucinous neoplasms (IPMNs) are overestimates, likely due to verification bias. MATERIALS AND METHODS: We tested for possible verification bias using simulation modeling techniques. First, in age-defined hypothetical cohorts of 10 million persons, we projected the frequency of pancreatic ductal adenocarcinoma (PDAC) arising from side-branch IPMNs over 5 years using published estimates of their prevalence (4.4%) and rate of malignant transformation (1.9%). Second, we projected the total number of PDAC cases in corresponding cohorts over the same time horizon using national cancer registry data. For each cohort, we determined whether the percentage of all PDAC cases that arose from side-branch IPMNs (i.e., side-branch IPMN-associated PDAC cases) was clinically plausible using an upper limit of 10% to define plausibility, as estimated from the literature. Model assumptions and parameter uncertainty were evaluated in sensitivity analysis. RESULTS: Across all cohorts, percentages of side-branch IPMN-associated PDACs greatly exceeded 10%. In the base case (mean age = 55.7 years), 80% of PDAC cases arose from side-branch IPMNs (7877/9786). In the oldest cohort evaluated (mean age = 75 years), this estimate was 76% (14,227/18,714). In a secondary analysis, we found that if an upper limit threshold of 10% for side-branch IPMN-associated PDAC was imposed, the model-predicted rate of malignancy for side-branch IPMNs would be less than 0.24% over a 5-year time horizon, substantially lower than most literature-based estimates. CONCLUSION: Our results suggest that reported malignancy risks associated with side-branch IPMNs are likely to be overestimates and imply the presence of verification bias.

Approximate confidence intervals for the likelihood ratios of a binary diagnostic test in the presence of partial disease verification.

The classic parameters used to assess the accuracy of a binary diagnostic test (BDT) are sensitivity and specificity. Other parameters used to describe the performance of a BDT are likelihood ratios (LRs). The LRs depend on the sensitivity and the specificity of the diagnostic test, and they reflect how much greater the probability of a positive or negative diagnostic test result for individuals with the disease than that for the individuals without the disease. In this study, several confidence intervals are studied for the LRs of a BDT in the presence of missing data. Two confidence intervals were studied through the method of maximum likelihood and seven confidence intervals were studied by applying the multiple imputation by chained equations method. A program in R software has been written that allows us to solve the estimation problem posed. The results obtained have been applied to the two real examples.

Determining sample size for a binary diagnostic test in the presence of verification bias.

To compare a new binary diagnostic test with the gold standard, sensitivity and specificity are the two common measurements used to evaluate the new test. When not all the patients are verified by the gold standard due to time, budget, or cost considerations, several approaches have been proposed to compute sample size for such studies under the assumption of missing completely at random. However, the majority of them are based on asymptotic approaches that generally do not guarantee the type I and II error rates, and the remaining approaches use exact binomial distributions in sample size calculation but only the verified samples are used. In this article, for a study with verification bias, we propose computing exact sample sizes by using all the samples. The proposed approach is compared with the existing exact approach that compute sample size by using verified samples only, and the results show that the proposed approach requires fewer participants than the competitor.

Inverse probability weighting estimation of the volume under the ROC surface in the presence of verification bias.

In diagnostic medicine, the volume under the receiver operating characteristic (ROC) surface (VUS) is a commonly used index to quantify the ability of a continuous diagnostic test to discriminate between three disease states. In practice, verification of the true disease status may be performed only for a subset of subjects under study since the verification procedure is invasive, risky, or expensive. The selection for disease examination might depend on the results of the diagnostic test and other clinical characteristics of the patients, which in turn can cause bias in estimates of the VUS. This bias is referred to as verification bias. Existing verification bias correction in three-way ROC analysis focuses on ordinal tests. We propose verification bias-correction methods to construct ROC surface and estimate the VUS for a continuous diagnostic test, based on inverse probability weighting. By applying U-statistics theory, we develop asymptotic properties for the estimator. A Jackknife estimator of variance is also derived. Extensive simulation studies are performed to evaluate the performance of the new estimators in terms of bias correction and variance. The proposed methods are used to assess the ability of a biomarker to accurately identify stages of Alzheimer's disease.

Bayesian ROC curve estimation under verification bias.

Receiver operating characteristic (ROC) curve has been widely used in medical science for its ability to measure the accuracy of diagnostic tests under the gold standard. However, in a complicated medical practice, a gold standard test can be invasive, expensive, and its result may not always be available for all the subjects under study. Thus, a gold standard test is implemented only when it is necessary and possible. This leads to the so-called 'verification bias', meaning that subjects with verified disease status (also called label) are not selected in a completely random fashion. In this paper, we propose a new Bayesian approach for estimating an ROC curve based on continuous data following the popular semiparametric binormal model in the presence of verification bias. By using a rank-based likelihood, and following Gibbs sampling techniques, we compute the posterior distribution of the binormal parameters intercept and slope, as well as the area under the curve by imputing the missing labels within Markov Chain Monte-Carlo iterations. Consistency of the resulting posterior under mild conditions is also established. We compare the new method with other comparable methods and conclude that our estimator performs well in terms of accuracy.

Colorectal Cancer Screening by Fecal Immunochemical Tests (FIT): Considerations on Sampling and Markers (Hb and Hb/Hp Complex) of Fecal Occult Blood (FOB).

BACKGROUND/AIM: Formal demonstration of the efficacy of colorectal cancer (CRC) screening by fecal immunochemical tests (FITs) in reducing CRC incidence and mortality is still missing. The aim of this study was to analyze the impact of sampling and FIT marker in the recently implemented CRC screening program in Finland. PATIENTS AND METHODS: Because only the index test [FIT hemoglobin (Hb)]-positive subjects are verified by the reference test (colonoscopy), the new screening program is subject to verification bias that precludes estimating the diagnostic accuracy (DA) indicators. A previously published study (5) with 100% biopsy verification of colonoscopy referral subjects (called validation cohort, n=300) was used to derive these missing DA estimates. Two points of concern were addressed: i) only one-day sample tested, and ii) only the Hb component (but not Hb/Hp complex) was analyzed by FIT. RESULTS: The estimated DA of one-sample testing for Hb in the screening setting had a very low sensitivity (SE) (12.5%; 95%CI=12.3-12.7) for adenomas, with AUC=0.560 (for CRC, AUC=0.950). Testing three samples for Hb improved SE to 19.4% (95%CI=19.1-19.7%) but had little effect on overall DA (AUC=0.590). For adenomas, one-sample testing for Hb and Hb/Hp complex provided higher SE than three-sample testing for Hb (SE 20.6%; 95%CI=20.3-21.0), and the best SE was reached when two samples were tested for Hb and Hb/Hp complex (SE 47.5%; 95%CI=46.9-48.1%) (AUC=0.730). CONCLUSION: The strategy of the current CRC screening could be significantly improved by testing two consecutive samples by Hb and Hb/Hp complex, instead of stand-alone Hb testing of one sample.

Development and Validation of Algorithms to Identify COVID-19 Patients Using a US Electronic Health Records Database: A Retrospective Cohort Study.

Introduction: In order to identify and evaluate candidate algorithms to detect COVID-19 cases in an electronic health record (EHR) database, this study examined and compared the utilization of acute respiratory disease codes from February to August 2020 versus the corresponding time period in the 3 years preceding. Methods: De-identified EHR data were used to identify codes of interest for candidate algorithms to identify COVID-19 patients. The number and proportion of patients who received a SARS-CoV-2 reverse transcriptase polymerase chain reaction (RT-PCR) within ±10 days of the occurrence of the diagnosis code and patients who tested positive among those with a test result were calculated, resulting in 11 candidate algorithms. Sensitivity, specificity, and likelihood ratios assessed the candidate algorithms by clinical setting and time period. We adjusted for potential verification bias by weighting by the reciprocal of the estimated probability of verification. Results: From January to March 2020, the most commonly used diagnosis codes related to COVID-19 diagnosis were R06 (dyspnea) and R05 (cough). On or after April 1, 2020, the code with highest sensitivity for COVID-19, U07.1, had near perfect adjusted sensitivity (1.00 [95% CI 1.00, 1.00]) but low adjusted specificity (0.32 [95% CI 0.31, 0.33]) in hospitalized patients. Discussion: Algorithms based on the U07.1 code had high sensitivity among hospitalized patients, but low specificity, especially after April 2020. None of the combinations of ICD-10-CM codes assessed performed with a satisfactory combination of high sensitivity and high specificity when using the SARS-CoV-2 RT-PCR as the reference standard.