An evaluation of computational methods for aggregate data meta-analyses of diagnostic test accuracy studies.

BACKGROUND: A Generalized Linear Mixed Model (GLMM) is recommended to meta-analyze diagnostic test accuracy studies (DTAs) based on aggregate or individual participant data. Since a GLMM does not have a closed-form likelihood function or parameter solutions, computational methods are conventionally used to approximate the likelihoods and obtain parameter estimates. The most commonly used computational methods are the Iteratively Reweighted Least Squares (IRLS), the Laplace approximation (LA), and the Adaptive Gauss-Hermite quadrature (AGHQ). Despite being widely used, it has not been clear how these computational methods compare and perform in the context of an aggregate data meta-analysis (ADMA) of DTAs. METHODS: We compared and evaluated the performance of three commonly used computational methods for GLMM - the IRLS, the LA, and the AGHQ, via a comprehensive simulation study and real-life data examples, in the context of an ADMA of DTAs. By varying several parameters in our simulations, we assessed the performance of the three methods in terms of bias, root mean squared error, confidence interval (CI) width, coverage of the 95% CI, convergence rate, and computational speed. RESULTS: For most of the scenarios, especially when the meta-analytic data were not sparse (i.e., there were no or negligible studies with perfect diagnosis), the three computational methods were comparable for the estimation of sensitivity and specificity. However, the LA had the largest bias and root mean squared error for pooled sensitivity and specificity when the meta-analytic data were sparse. Moreover, the AGHQ took a longer computational time to converge relative to the other two methods, although it had the best convergence rate. CONCLUSIONS: We recommend practitioners and researchers carefully choose an appropriate computational algorithm when fitting a GLMM to an ADMA of DTAs. We do not recommend the LA for sparse meta-analytic data sets. However, either the AGHQ or the IRLS can be used regardless of the characteristics of the meta-analytic data.

An empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant data.

BACKGROUND: Selective reporting of results from only well-performing cut-offs leads to biased estimates of accuracy in primary studies of questionnaire-based screening tools and in meta-analyses that synthesize results. Individual participant data meta-analysis (IPDMA) of sensitivity and specificity at each cut-off via bivariate random-effects models (BREMs) can overcome this problem. However, IPDMA is laborious and depends on the ability to successfully obtain primary datasets, and BREMs ignore the correlation between cut-offs within primary studies. METHODS: We compared the performance of three recent multiple cut-off models developed by Steinhauser et al., Jones et al., and Hoyer and Kuss, that account for missing cut-offs when meta-analyzing diagnostic accuracy studies with multiple cut-offs, to BREMs fitted at each cut-off. We used data from 22 studies of the accuracy of the Edinburgh Postnatal Depression Scale (EPDS; 4475 participants, 758 major depression cases). We fitted each of the three multiple cut-off models and BREMs to a dataset with results from only published cut-offs from each study (published data) and an IPD dataset with results for all cut-offs (full IPD data). We estimated pooled sensitivity and specificity with 95% confidence intervals (CIs) for each cut-off and the area under the curve. RESULTS: Compared to the BREMs fitted to the full IPD data, the Steinhauser et al., Jones et al., and Hoyer and Kuss models fitted to the published data produced similar receiver operating characteristic curves; though, the Hoyer and Kuss model had lower area under the curve, mainly due to estimating slightly lower sensitivity at lower cut-offs. When fitting the three multiple cut-off models to the full IPD data, a similar pattern of results was observed. Importantly, all models had similar 95% CIs for sensitivity and specificity, and the CI width increased with cut-off levels for sensitivity and decreased with an increasing cut-off for specificity, even the BREMs which treat each cut-off separately. CONCLUSIONS: Multiple cut-off models appear to be the favorable methods when only published data are available. While collecting IPD is expensive and time consuming, IPD can facilitate subgroup analyses that cannot be conducted with published data only.

Accuracy of the Patient Health Questionnaire-9 for screening to detect major depression: updated systematic review and individual participant data meta-analysis.

OBJECTIVE: To update a previous individual participant data meta-analysis and determine the accuracy of the Patient Health Questionnaire-9 (PHQ-9), the most commonly used depression screening tool in general practice, for detecting major depression overall and by study or participant subgroups. DESIGN: Systematic review and individual participant data meta-analysis. DATA SOURCES: Medline, Medline In-Process, and Other Non-Indexed Citations via Ovid, PsycINFO, Web of Science searched through 9 May 2018. REVIEW METHODS: Eligible studies administered the PHQ-9 and classified current major depression status using a validated semistructured diagnostic interview (designed for clinician administration), fully structured interview (designed for lay administration), or the Mini International Neuropsychiatric Interview (MINI; a brief interview designed for lay administration). A bivariate random effects meta-analytic model was used to obtain point and interval estimates of pooled PHQ-9 sensitivity and specificity at cut-off values 5-15, separately, among studies that used semistructured diagnostic interviews (eg, Structured Clinical Interview for Diagnostic and Statistical Manual), fully structured interviews (eg, Composite International Diagnostic Interview), and the MINI. Meta-regression was used to investigate whether PHQ-9 accuracy correlated with reference standard categories and participant characteristics. RESULTS: Data from 44 503 total participants (27 146 additional from the update) were obtained from 100 of 127 eligible studies (42 additional studies; 79% eligible studies; 86% eligible participants). Among studies with a semistructured interview reference standard, pooled PHQ-9 sensitivity and specificity (95% confidence interval) at the standard cut-off value of ≥10, which maximised combined sensitivity and specificity, were 0.85 (0.79 to 0.89) and 0.85 (0.82 to 0.87), respectively. Specificity was similar across reference standards, but sensitivity in studies with semistructured interviews was 7-24% (median 21%) higher than with fully structured reference standards and 2-14% (median 11%) higher than with the MINI across cut-off values. Across reference standards and cut-off values, specificity was 0-10% (median 3%) higher for men and 0-12 (median 5%) higher for people aged 60 or older. CONCLUSIONS: Researchers and clinicians could use results to determine outcomes, such as total number of positive screens and false positive screens, at different PHQ-9 cut-off values for different clinical settings using the knowledge translation tool at www.depressionscreening100.com/phq. STUDY REGISTRATION: PROSPERO CRD42014010673.

Validation of the COVID-19 Fears Questionnaires for Chronic Medical Conditions: A Scleroderma Patient-centered Intervention Network COVID-19 Cohort study.

OBJECTIVE: Fear associated with medical vulnerability should be considered when assessing mental health among individuals with chronic medical conditions during the COVID-19 pandemic. The objective was to develop and validate the COVID-19 Fears Questionnaire for Chronic Medical Conditions. METHODS: Fifteen initial items were generated based on suggestions from 121 people with the chronic autoimmune disease systemic sclerosis (SSc; scleroderma). Patients in a COVID-19 SSc cohort completed items between April 9 and 27, 2020. Exploratory factor analysis (EFA) and item analysis were used to select items for inclusion. Cronbach's alpha and Pearson correlations were used to evaluate internal consistency reliability and convergent validity. Factor structure was confirmed with confirmatory factor analysis (CFA) in follow-up data collection two weeks later. RESULTS: 787 participants completed baseline measures; 563 of them completed the follow-up assessment. Ten of 15 initial items were included in the final questionnaire. EFA suggested that a single dimension explained the data reasonably well. There were no indications of floor or ceiling effects. Cronbach's alpha was 0.91. Correlations between the COVID-19 Fears Questionnaire and measures of anxiety (r = 0.53), depressive symptoms (r = 0.44), and perceived stress (r = 0.50) supported construct validity. CFA supported the single-factor structure (χ2(35) = 311.2, p < 0.001, Tucker-Lewis Index = 0.97, Comparative Fit Index = 0.96, Root Mean Square Error of Approximation = 0.12). CONCLUSION: The COVID-19 Fears Questionnaire for Chronic Medical Conditions can be used to assess fear among people at risk due to pre-existing medical conditions during the COVID-19 pandemic.

Robust bivariate random-effects model for accommodating outlying and influential studies in meta-analysis of diagnostic test accuracy studies.

Due to the inevitable inter-study correlation between test sensitivity (Se) and test specificity (Sp), mostly because of threshold variability, hierarchical or bivariate random-effects models are widely used to perform a meta-analysis of diagnostic test accuracy studies. Conventionally, these models assume that the random-effects follow the bivariate normal distribution. However, the inference made using the well-established bivariate random-effects models, when outlying and influential studies are present, may lead to misleading conclusions, since outlying or influential studies can extremely influence parameter estimates due to their disproportional weight. Therefore, we developed a new robust bivariate random-effects model that accommodates outlying and influential observations and gives robust statistical inference by down-weighting the effect of outlying and influential studies. The marginal model and the Monte Carlo expectation-maximization algorithm for our proposed model have been derived. A simulation study has been carried out to validate the proposed method and compare it against the standard methods. Regardless of the parameters varied in our simulations, the proposed model produced robust point estimates of Se and Sp compared to the standard models. Moreover, our proposed model resulted in precise estimates as it yielded the narrowest confidence intervals. The proposed model also generated a similar point and interval estimates of Se and Sp as the standard models when there are no outlying and influential studies. Two published meta-analyses have also been used to illustrate the methods.

Skew-normal random-effects model for meta-analysis of diagnostic test accuracy (DTA) studies.

Hierarchical models are recommended for meta-analyzing diagnostic test accuracy (DTA) studies. The bivariate random-effects model is currently widely used to synthesize a pair of test sensitivity and specificity using logit transformation across studies. This model assumes a bivariate normal distribution for the random-effects. However, this assumption is restrictive and can be violated. When the assumption fails, inferences could be misleading. In this paper, we extended the current bivariate random-effects model by assuming a flexible bivariate skew-normal distribution for the random-effects in order to robustly model logit sensitivities and logit specificities. The marginal distribution of the proposed model is analytically derived so that parameter estimation can be performed using standard likelihood methods. The method of weighted-average is adopted to estimate the overall logit-transformed sensitivity and specificity. An extensive simulation study is carried out to investigate the performance of the proposed model compared to other standard models. Overall, the proposed model performs better in terms of confidence interval width of the average logit-transformed sensitivity and specificity compared to the standard bivariate linear mixed model and bivariate generalized linear mixed model. Simulations have also shown that the proposed model performed better than the well-established bivariate linear mixed model in terms of bias and comparable with regards to the root mean squared error (RMSE) of the between-study (co)variances. The proposed method is also illustrated using a published meta-analysis data.

Statistical methods for detecting outlying and influential studies in meta-analysis of diagnostic test accuracy studies.

Bivariate random-effects models are currently widely used to synthesize pairs of test sensitivity and specificity across studies. Inferences drawn based on these models may be distorted in the presence of outlying or influential studies. Currently, subjective methods such as inspection of forest plots are used to identify outlying studies in meta-analysis of diagnostic test accuracy studies. We proposed objective methods based on solid statistical reasoning for identifying outlying and/or influential studies. The proposed methods have been validated using simulation study and illustrated on two published meta-analysis data. Our methods outperform and neglect the subjectivity of the currently used ad hoc methods. The proposed methods can be used as a sensitivity analysis tool concurrently with the current bivariate random-effects models or as a preliminary analysis tool for robust models that accommodate outlying and/or influential studies in meta-analysis of diagnostic test accuracy studies.

Bivariate random-effects meta-analysis models for diagnostic test accuracy studies using arcsine-based transformations.

Diagnostic or screening tests are widely used in medical fields to classify patients according to their disease status. Several statistical models for meta-analysis of diagnostic test accuracy studies have been developed to synthesize test sensitivity and specificity of a diagnostic test of interest. Because of the correlation between test sensitivity and specificity, modeling the two measures using a bivariate model is recommended. In this paper, we extend the current standard bivariate linear mixed model (LMM) by proposing two variance-stabilizing transformations: the arcsine square root and the Freeman-Tukey double arcsine transformation. We compared the performance of the proposed methods with the standard method through simulations using several performance measures. The simulation results showed that our proposed methods performed better than the standard LMM in terms of bias, root mean square error, and coverage probability in most of the scenarios, even when data were generated assuming the standard LMM. We also illustrated the methods using two real data sets.

Galactomannan, ß-D-Glucan, and Polymerase Chain Reaction-Based Assays for the Diagnosis of Invasive Fungal Disease in Pediatric Cancer and Hematopoietic Stem Cell Transplantation: A Systematic Review and Meta-Analysis.

We systematically reviewed and analyzed the available data for galactomannan (GM), ß-D-glucan (BG), and polymerase chain reaction (PCR)-based assays to detect invasive fungal disease (IFD) in patients with pediatric cancer or undergoing hematopoietic stem cell transplantation when used as screening tools during immunosuppression or as diagnostic tests in patients presenting with symptoms such as fever during neutropenia (FN). Of 1532 studies screened, 25 studies reported on GM (n = 19), BG (n = 3), and PCR (n = 11). All fungal biomarkers demonstrated highly variable sensitivity, specificity, and positive predictive values, and these were generally poor in both clinical settings. GM negative predictive values were high, ranging from 85% to 100% for screening and 70% to 100% in the diagnostic setting, but failure to identify non-Aspergillus molds limits its usefulness. Future work could focus on the usefulness of combinations of fungal biomarkers in pediatric cancer and HSCT.