How to Critically Appraise and Interpret Systematic Reviews and Meta-Analyses of Diagnostic Accuracy: A User Guide.

Systematic reviews of diagnostic accuracy studies can provide the best available evidence to inform decisions regarding the use of a diagnostic test. In this guide, the authors provide a practical approach for clinicians to appraise diagnostic accuracy systematic reviews and apply their results to patient care. The first step is to identify an appropriate systematic review with a research question matching the clinical scenario. The user should evaluate the rigor of the review methods to evaluate its credibility (Did the review use clearly defined eligibility criteria, a comprehensive search strategy, structured data collection, risk of bias and applicability appraisal, and appropriate meta-analysis methods?). If the review is credible, the next step is to decide whether the diagnostic performance is adequate for clinical use (Do sensitivity and specificity estimates exceed the threshold that makes them useful in clinical practice? Are these estimates sufficiently precise? Is variability in the estimates of diagnostic accuracy across studies explained?). Diagnostic accuracy systematic reviews that are judged to be credible and provide diagnostic accuracy estimates with sufficient certainty and relevance are the most useful to inform patient care. This review discusses comparative, noncomparative, and emerging approaches to systematic reviews of diagnostic accuracy using a clinical scenario and examples based on recent publications.

Diagnostic Accuracy of MRI for Solid Renal Masses: A Systematic Review and Meta-analysis.

BACKGROUND: Biparametric (bp)-MRI and multiparametric (mp)-MRI may improve the diagnostic accuracy of renal mass histology. PURPOSE: To evaluate the available evidence on the diagnostic accuracy of bp-MRI and mp-MRI for solid renal masses in differentiating malignant from benign, aggressive from indolent, and clear cell renal cell carcinoma (ccRCC) from other histology. STUDY TYPE: Systematic review. POPULATION: MEDLINE, EMBASE, and CENTRAL up to January 11, 2022 were searched. FIELD STRENGTH/SEQUENCE: 1.5 or 3 Tesla. ASSESSMENT: Eligible studies evaluated the accuracy of MRI (with at least two sequences: T2, T1, dynamic contrast and diffusion-weighted imaging) for diagnosis of solid renal masses in adult patients, using histology as reference standard. Risk of bias and applicability were assessed using QUADAS-2. STATISTICAL TESTS: Meta-analysis using a bivariate logitnormal random effects model. RESULTS: We included 10 studies (1239 masses from approximately 1200 patients). The risk of bias was high in three studies, unclear in five studies and low in two studies. The diagnostic accuracy of malignant (vs. benign) masses was assessed in five studies (64% [179/281] malignant). The summary estimate of sensitivity was 95% (95% confidence interval [CI]: 77%-99%), and specificity was 63% (95% CI: 46%-77%). No study assessed aggressive (vs. indolent) masses. The diagnostic accuracy of ccRCC (vs. other subtypes) was evaluated in six studies (47% [455/971] ccRCC): the summary estimate of sensitivity was 85% (95% CI: 77%-90%) and specificity was 77% (95% CI: 73%-81%). DATA CONCLUSION: Our study reveals deficits in the available evidence on MRI for diagnosis of renal mass histology. The number of studies was limited, at unclear/high risk of bias, with heterogeneous definitions of solid masses, imaging techniques, diagnostic criteria, and outcome measures. EVIDENCE LEVEL: 3 TECHNICAL EFFICACY: Stage 2.

Evaluating the Impact of Peer Review on the Completeness of Reporting in Imaging Diagnostic Test Accuracy Research.

BACKGROUND: Despite the nearly ubiquitous reported use of peer review among reputable medical journals, there is limited evidence to support the use of peer review to improve the quality of biomedical research and in particular, imaging diagnostic test accuracy (DTA) research. PURPOSE: To evaluate whether peer review of DTA studies published by imaging journals is associated with changes in completeness of reporting, transparency for risk of bias assessment, and spin. STUDY TYPE: Retrospective cross-sectional study. STUDY SAMPLE: Cross-sectional study of articles published in Journal of Magnetic Resonance Imaging (JMRI), Canadian Association of Radiologists Journal (CARJ), and European Radiology (EuRad) before March 31, 2020. ASSESSMENT: Initial submitted and final versions of manuscripts were evaluated for completeness of reporting using the Standards for Reporting Diagnostic Accuracy Studies (STARD) 2015 and STARD for Abstracts guidelines, transparency of reporting for risk of bias assessment based on Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2), and actual and potential spin using modified published criteria. STATISTICAL TESTS: Two-tailed paired t-tests and paired Wilcoxon signed-rank tests were used for comparisons. A P value <0.05 was considered to be statistically significant. RESULTS: We included 84 diagnostic accuracy studies accepted by three journals between 2014 and 2020 (JMRI = 30, CARJ = 23, and EuRad = 31) of the 692 which were screened. Completeness of reporting according to STARD 2015 increased significantly between initial submissions and final accepted versions (average reported items: 16.67 vs. 17.47, change of 0.80 [95% confidence interval 0.25-1.17]). No significant difference was found for the reporting of STARD for Abstracts (5.28 vs. 5.25, change of -0.03 [-0.15 to 0.11], P = 0.74), QUADAS-2 (6.08 vs. 6.11, change of 0.03 [-1.00 to 0.50], P = 0.92), actual "spin" (2.36 vs. 2.40, change of 0.04 [0.00 to 1.00], P = 0.39) or potential "spin" (2.93 vs. 2.81, change of -0.12 [-1.00 to 0.00], P = 0.23) practices. CONCLUSION: Peer review is associated with a marginal improvement in completeness of reporting in published imaging DTA studies, but not with improvement in transparency for risk of bias assessment or reduction in spin. LEVEL OF EVIDENCE: 3 TECHNICAL EFFICACY STAGE: 1.

Association of Accuracy, Conclusions, and Reporting Completeness With Acceptance by Radiology Conferences and Journals.

BACKGROUND: Preferential publication of studies with positive findings can lead to overestimation of diagnostic test accuracy (i.e. publication bias). Understanding the contribution of the editorial process to publication bias could inform interventions to optimize the evidence guiding clinical decisions. PURPOSE/HYPOTHESIS: To evaluate whether accuracy estimates, abstract conclusion positivity, and completeness of abstract reporting are associated with acceptance to radiology conferences and journals. STUDY TYPE: Meta-research. POPULATION: Abstracts submitted to radiology conferences (European Society of Gastrointestinal and Abdominal Radiology (ESGAR) and International Society for Magnetic Resonance in Medicine (ISMRM)) from 2008 to 2018 and manuscripts submitted to radiology journals (Radiology, Journal of Magnetic Resonance Imaging [JMRI]) from 2017 to 2018. Primary clinical studies evaluating sensitivity and specificity of a diagnostic imaging test in humans with available editorial decisions were included. ASSESSMENT: Primary variables (Youden's index [YI > 0.8 vs. <0.8], abstract conclusion positivity [positive vs. neutral/negative], number of reported items on the Standards for Reporting of Diagnostic Accuracy Studies [STARD] for Abstract guideline) and confounding variables (prospective vs. retrospective/unreported, sample size, study duration, interobserver agreement assessment, subspecialty, modality) were extracted. STATISTICAL TESTS: Multivariable logistic regression to obtain adjusted odds ratio (OR) as a measure of the association between the primary variables and acceptance by radiology conferences and journals; 95% confidence intervals (CIs) and P-values were obtained; the threshold for statistical significance was P < 0.05. RESULTS: A total of 1000 conference abstracts (500 ESGAR and 500 ISMRM) and 1000 journal manuscripts (505 Radiology and 495 JMRI) were included. Conference abstract acceptance was not significantly associated with YI (adjusted OR = 0.97 for YI > 0.8; CI = 0.70-1.35), conclusion positivity (OR = 1.21 for positive conclusions; CI = 0.75-1.90) or STARD for Abstracts adherence (OR = 0.96 per unit increase in reported items; CI = 0.82-1.18). Manuscripts with positive abstract conclusions were less likely to be accepted by radiology journals (OR = 0.45; CI = 0.24-0.86), while YI (OR = 0.85; CI = 0.56-1.29) and STARD for Abstracts adherence (OR = 1.06; CI = 0.87-1.30) showed no significant association. Positive conclusions were present in 86.7% of submitted conference abstracts and 90.2% of journal manuscripts. DATA CONCLUSION: Diagnostic test accuracy studies with positive findings were not preferentially accepted by the evaluated radiology conferences or journals. EVIDENCE LEVEL: 3 TECHNICAL EFFICACY: Stage 2.

Tweeting Bias in Diagnostic Test Accuracy Research: Does Title or Conclusion Positivity Influence Dissemination?

PURPOSE: To examine if tweeting bias exists within imaging literature by determining if diagnostic test accuracy (DTA) studies with positive titles or conclusions are tweeted more than non-positive studies. METHODS: DTA studies published between October 2011 to April 2016 were included. Positivity of titles and conclusions were assessed independently and in duplicate, with disagreements resolved by consensus. A negative binomial regression analysis controlling for confounding variables was performed to assess the relationship between title or conclusion positivity and tweets an article received in the 100 days post-publication. RESULTS: 354 DTA studies were included. Twenty-four (7%) titles and 300 (85%) conclusions were positive (or positive with qualifier); 1 (0.3%) title and 23 (7%) conclusions were negative; and 329 (93%) titles and 26 (7%) conclusions were neutral. Studies with positive, negative, and neutral titles received a mean of 0.38, 0.00, and 0.45 tweets per study; while those with positive, negative, and neutral conclusions received a mean of 0.44, 0.61, and 0.38 tweets per study. Regression coefficients were -0.05 (SE 0.46) for positive relative to non-positive titles, and -0.09 (SE 0.31) for positive relative to non-positive conclusions. The positivity of the title (P = 0.91) or conclusion (P = 0.76) was not significantly associated with the number of tweets an article received. CONCLUSIONS: The positivity of the title or conclusion for DTA studies does not influence the amount of tweets it receives suggesting that tweet bias is not present among imaging diagnostic accuracy studies. Study protocol available at https://osf.io/hdk2m/.

Reporting Bias in Imaging Diagnostic Test Accuracy Studies: Are Studies With Positive Conclusions or Titles Submitted and Published Faster?

OBJECTIVE: The purpose of this study is to evaluate whether imaging diagnostic test accuracy (DTA) studies with positive conclusions or titles have a shorter time to publication than those with nonpositive (i.e., negative or neutral) conclusions or titles. MATERIALS AND METHODS: We included primary imaging DTA studies from systematic reviews published in 2015. The conclusion and title of each study were extracted, and their positivity was classified independently in duplicate. The time from study completion to publication was extracted and calculated. A Cox regression model was used to evaluate associations of conclusion and title positivity with time to publication, with adjustment made for potentially confounding variables. RESULTS: A total of 774 imaging DTA studies were included; time from study completion to publication could be calculated for 516 studies. The median time from completion to publication was 18 months (interquartile range, 13-26 months) for the 413 studies with positive conclusions, 23 months (interquartile range, 16-33 months) for the 63 studies with neutral conclusions, and 25 months (interquartile range, 15-38 months) for the 40 studies with negative conclusions. A positive conclusion was associated with a shorter time from study completion to publication compared with a non-positive conclusion (hazard ratio, 1.31; 95% CI, 1.02-1.68). Of all included studies, 39 (5%) had positive titles, 731 (94%) had neutral titles, and 4 (< 1%) had negative titles. Positive titles were not significantly associated with a shorter time to study publication (hazard ratio, 1.12; 95% CI, 0.75-1.69). CONCLUSION: Positive conclusions (but not titles) were associated with a shorter time from study completion to publication. This finding may contribute to an overrepresentation of positive results in the imaging DTA literature.

Overinterpretation of Research Findings: Evaluation of "Spin" in Systematic Reviews of Diagnostic Accuracy Studies in High-Impact Factor Journals.

BACKGROUND: To compare the frequency of "spin" in systematic reviews of diagnostic accuracy studies in high-impact journals with the frequency a previously assessed series of reviews. METHODS: Medline was searched from January 2010 to January 2019. Systematic reviews of diagnostic accuracy studies were included if they reported a meta-analysis and were published in a journal with an impact factor >5. Two investigators independently scored each included systematic review for positivity of conclusions and for actual and potential overinterpretation practices. RESULTS: Of 137 included systematic reviews, actual overinterpretation was present in ≥1 form in the abstract in 63 (46%) and in the full-text report in 52 (38%); 108 (79%) contained a form of potential overinterpretation. Compared with the previously assessed series (reviews published 2015-2016), reviews in this series were less likely to contain ≥1 form of actual overinterpretation in the abstract and full-text report or ≥1 form of potential overinterpretation (P < 0.001 for all comparisons). The significance of these comparisons did not persist for actual overinterpretation in sensitivity analysis in which Cochrane systematic reviews were removed. Reviews published in the Cochrane Database of Systematic Reviews were less likely to contain actual overinterpretation in the abstract or the full-text report than reviews in other high-impact journals (P < 0.001 for both comparisons). CONCLUSIONS: Reviews of diagnostic accuracy studies in high-impact journals are less likely to contain overinterpretation or spin. This difference is largely due to the reviews published in the Cochrane Database of Systematic Reviews, which contain spin less often than reviews published in other high-impact journals.

Diagnostic accuracy of dual-energy computed tomography (DECT) to differentiate uric acid from non-uric acid calculi: systematic review and meta-analysis.

BACKGROUND: Uric acid stone diagnosis is presently done primarily with in vitro analysis of stones. In vivo diagnosis with dual-energy CT (DECT) would allow earlier initiation of therapy with urine alkalinization and avoid surgical intervention. OBJECTIVE: To evaluate if DECT, using stone analysis as reference standard, is sufficiently accurate to replace stone analysis for diagnosis of uric acid stones. METHODS: Original studies in patients with urolithiasis examined with DECT with stone analysis as the reference standard were eligible for inclusion. MEDLINE (1946-2018), Embase (1947-2018), CENTRAL (August 2018), and multiple urology and radiology conferences were searched. QUADAS-2 was used to assess risk of bias and applicability. Meta-analyses were performed using a bivariate random-effects model. RESULTS: A total of 21 studies (1105 patients, 1442 stones) were included. Fourteen studies containing 662 patients (944 stones) were analyzed in the uric acid dominant target condition (majority of stone composition uric acid): mean sensitivity was 0.88 (95% CI 0.79-0.93) and specificity 0.98 (95% CI 0.96-0.99). Thirteen studies (674 patients, 760 stones) were analyzed in the uric acid-containing target condition (< majority of stone composition uric acid): mean sensitivity was 0.82 (95% CI 0.73-0.89) and specificity 0.97 (95% CI 0.94-0.98). Meta-regression showed no significant variability in test accuracy. Two studies had one or more domains at high risk of bias and there were no concerns regarding applicability. CONCLUSION: DECT is an accurate replacement test for diagnosis of uric acid calculi in vivo, such that stone analysis could be replaced in the diagnostic pathway. This would enable earlier initiation of urine alkalinization. KEY POINTS: • DECT for uric acid dominant stones has sensitivity of 0.88 (95% CI 0.79-0.93) and specificity of 0.98 (95% CI 0.96-0.99); uric acid-containing stones had mean sensitivity of 0.82 (95% CI 0.73-0.89) and specificity of 0.97 (95% CI 0.94-0.98). • Meta-regression did not identify any variables (study design, reference standard, dual-energy CT type, dose, risk of bias) that influenced test accuracy. • Only 2 of the 21 included studies had 1 or more domain considered to be at high risk of bias with the majority of domains considered at low risk of bias; there were no concerns regarding applicability in any of the included studies.

Publication bias in diagnostic imaging: conference abstracts with positive conclusions are more likely to be published.

OBJECTIVE: To evaluate whether imaging diagnostic test accuracy conference abstracts with positive conclusions or titles are more likely to reach full-text publication than those with negative (or neutral) conclusions or titles. METHODS: Diagnostic accuracy research abstracts were included if they were presented at the 2011 or 2012 Radiological Society of North America conference. Full-text publication status at 5 years post conference abstract submission was determined. Conclusion and title positivity of conference abstracts were extracted, as well as potential confounding factors. The associations of conclusion and title positivity with publication status at 5 years post conference abstract submission were assessed using a multivariable logistic regression model. Conditional odds ratios were calculated to express the strength of associations, adjusting for the confounders. RESULTS: In total, 282/400 (71%) of included conference abstracts reached full-text publication. A total of 246 out of 337 (74%) conference abstracts with positive conclusions resulted in full-text publications, compared with 26/48 (54%) with neutral conclusions and 5/15 (33%) with negative conclusions. In multivariable logistic regression, conclusion positivity was significantly associated with full-text publication (odds ratio 3.6; 95% CI 1.9-6.7 for conference abstracts with positive conclusions, compared with those with non-positive conclusions); this did not apply to title positivity (odds ratio 1.2; 95% CI 0.47-3.0). CONCLUSION: Imaging conference abstracts with positive conclusions were more likely to be published as full-text articles. Title positivity was not associated with publication. This preferential publication pattern may lead to an overrepresentation of positive studies in the literature. An overrepresentation of positive studies may contribute to inflated estimates of test accuracy and has the potential to adversely influence patient care. KEY POINTS: • Imaging diagnostic test accuracy conference abstracts with positive conclusions were more likely to be reported as full-text articles than those with non-positive conclusions. • The majority (75%) of imaging diagnostic test accuracy conference abstracts with positive conclusions were published, compared with only 53% and 33% with neutral and negative conclusions, respectively. • Conclusion positivity remained associated with the full-text publication of conference abstracts when controlling for multiple potential confounding variables.

Thoracic imaging tests for the diagnosis of COVID-19.

BACKGROUND: The respiratory illness caused by SARS-CoV-2 infection continues to present diagnostic challenges. Early research showed thoracic (chest) imaging to be sensitive but not specific in the diagnosis of coronavirus disease 2019 (COVID-19). However, this is a rapidly developing field and these findings need to be re-evaluated in the light of new research. This is the first update of this 'living systematic review'. This update focuses on people suspected of having COVID-19 and excludes studies with only confirmed COVID-19 participants. OBJECTIVES: To evaluate the diagnostic accuracy of thoracic imaging (computed tomography (CT), X-ray and ultrasound) in people with suspected COVID-19. SEARCH METHODS: We searched the COVID-19 Living Evidence Database from the University of Bern, the Cochrane COVID-19 Study Register, The Stephen B. Thacker CDC Library, and repositories of COVID-19 publications through to 22 June 2020. We did not apply any language restrictions. SELECTION CRITERIA: We included studies of all designs that recruited participants of any age group suspected to have COVID-19, and which reported estimates of test accuracy, or provided data from which estimates could be computed. When studies used a variety of reference standards, we retained the classification of participants as COVID-19 positive or negative as used in the study. DATA COLLECTION AND ANALYSIS: We screened studies, extracted data, and assessed the risk of bias and applicability concerns using the QUADAS-2 domain-list independently, in duplicate. We categorised included studies into three groups based on classification of index test results: studies that reported specific criteria for index test positivity (group 1); studies that did not report specific criteria, but had the test reader(s) explicitly classify the imaging test result as either COVID-19 positive or negative (group 2); and studies that reported an overview of index test findings, without explicitly classifying the imaging test as either COVID-19 positive or negative (group 3). We presented the results of estimated sensitivity and specificity using paired forest plots, and summarised in tables. We used a bivariate meta-analysis model where appropriate. We presented uncertainty of the accuracy estimates using 95% confidence intervals (CIs). MAIN RESULTS: We included 34 studies: 30 were cross-sectional studies with 8491 participants suspected of COVID-19, of which 4575 (54%) had a final diagnosis of COVID-19; four were case-control studies with 848 cases and controls in total, of which 464 (55%) had a final diagnosis of COVID-19. Chest CT was evaluated in 31 studies (8014 participants, 4224 (53%) cases), chest X-ray in three studies (1243 participants, 784 (63%) cases), and ultrasound of the lungs in one study (100 participants, 31 (31%) cases). Twenty-six per cent (9/34) of all studies were available only as preprints. Nineteen studies were conducted in Asia, 10 in Europe, four in North America and one in Australia. Sixteen studies included only adults, 15 studies included both adults and children and one included only children. Two studies did not report the ages of participants. Twenty-four studies included inpatients, four studies included outpatients, while the remaining six studies were conducted in unclear settings. The majority of included studies had a high or unclear risk of bias with respect to participant selection, index test, reference standard, and participant flow. For chest CT in suspected COVID-19 participants (31 studies, 8014 participants, 4224 (53%) cases) the sensitivity ranged from 57.4% to 100%, and specificity ranged from 0% to 96.0%. The pooled sensitivity of chest CT in suspected COVID-19 participants was 89.9% (95% CI 85.7 to 92.9) and the pooled specificity was 61.1% (95% CI 42.3 to 77.1). Sensitivity analyses showed that when the studies from China were excluded, the studies from other countries demonstrated higher specificity compared to the overall included studies. When studies that did not classify index tests as positive or negative for COVID-19 (group 3) were excluded, the remaining studies (groups 1 and 2) demonstrated higher specificity compared to the overall included studies. Sensitivity analyses limited to cross-sectional studies, or studies where at least two reverse transcriptase polymerase chain reaction (RT-PCR) tests were conducted if the first was negative, did not substantively alter the accuracy estimates. We did not identify publication status as a source of heterogeneity. For chest X-ray in suspected COVID-19 participants (3 studies, 1243 participants, 784 (63%) cases) the sensitivity ranged from 56.9% to 89.0% and specificity from 11.1% to 88.9%. The sensitivity and specificity of ultrasound of the lungs in suspected COVID-19 participants (1 study, 100 participants, 31 (31%) cases) were 96.8% and 62.3%, respectively. We could not perform a meta-analysis for chest X-ray or ultrasound due to the limited number of included studies. AUTHORS' CONCLUSIONS: Our findings indicate that chest CT is sensitive and moderately specific for the diagnosis of COVID-19 in suspected patients, meaning that CT may have limited capability in differentiating SARS-CoV-2 infection from other causes of respiratory illness. However, we are limited in our confidence in these results due to the poor study quality and the heterogeneity of included studies. Because of limited data, accuracy estimates of chest X-ray and ultrasound of the lungs for the diagnosis of suspected COVID-19 cases should be carefully interpreted. Future diagnostic accuracy studies should pre-define positive imaging findings, include direct comparisons of the various modalities of interest on the same participant population, and implement improved reporting practices. Planned updates of this review will aim to: increase precision around the accuracy estimates for chest CT (ideally with low risk of bias studies); obtain further data to inform accuracy of chest X-rays and ultrasound; and obtain data to further fulfil secondary objectives (e.g. 'threshold' effects, comparing accuracy estimates across different imaging modalities) to inform the utility of imaging along different diagnostic pathways.

Thoracic imaging tests for the diagnosis of COVID-19.

BACKGROUND: The diagnosis of infection by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) presents major challenges. Reverse transcriptase polymerase chain reaction (RT-PCR) testing is used to diagnose a current infection, but its utility as a reference standard is constrained by sampling errors, limited sensitivity (71% to 98%), and dependence on the timing of specimen collection. Chest imaging tests are being used in the diagnosis of COVID-19 disease, or when RT-PCR testing is unavailable. OBJECTIVES: To determine the diagnostic accuracy of chest imaging (computed tomography (CT), X-ray and ultrasound) in people with suspected or confirmed COVID-19. SEARCH METHODS: We searched the COVID-19 Living Evidence Database from the University of Bern, the Cochrane COVID-19 Study Register, and The Stephen B. Thacker CDC Library. In addition, we checked repositories of COVID-19 publications. We did not apply any language restrictions. We conducted searches for this review iteration up to 5 May 2020. SELECTION CRITERIA: We included studies of all designs that produce estimates of test accuracy or provide data from which estimates can be computed. We included two types of cross-sectional designs: a) where all patients suspected of the target condition enter the study through the same route and b) where it is not clear up front who has and who does not have the target condition, or where the patients with the target condition are recruited in a different way or from a different population from the patients without the target condition. When studies used a variety of reference standards, we included all of them. DATA COLLECTION AND ANALYSIS: We screened studies and extracted data independently, in duplicate. We also assessed the risk of bias and applicability concerns independently, in duplicate, using the QUADAS-2 checklist and presented the results of estimated sensitivity and specificity, using paired forest plots, and summarised in tables. We used a hierarchical meta-analysis model where appropriate. We presented uncertainty of the accuracy estimates using 95% confidence intervals (CIs). MAIN RESULTS: We included 84 studies, falling into two categories: studies with participants with confirmed diagnoses of COVID-19 at the time of recruitment (71 studies with 6331 participants) and studies with participants suspected of COVID-19 (13 studies with 1948 participants, including three case-control studies with 549 cases and controls). Chest CT was evaluated in 78 studies (8105 participants), chest X-ray in nine studies (682 COVID-19 cases), and chest ultrasound in two studies (32 COVID-19 cases). All evaluations of chest X-ray and ultrasound were conducted in studies with confirmed diagnoses only. Twenty-five per cent (21/84) of all studies were available only as preprints, 15/71 studies in the confirmed cases group and 6/13 of the studies in the suspected group. Among 71 studies that included confirmed cases, 41 studies had included symptomatic cases only, 25 studies had included cases regardless of their symptoms, five studies had included asymptomatic cases only, three of which included a combination of confirmed and suspected cases. Seventy studies were conducted in Asia, 2 in Europe, 2 in North America and one in South America. Fifty-one studies included inpatients while the remaining 24 studies were conducted in mixed or unclear settings. Risk of bias was high in most studies, mainly due to concerns about selection of participants and applicability. Among the 13 studies that included suspected cases, nine studies were conducted in Asia, and one in Europe. Seven studies included inpatients while the remaining three studies were conducted in mixed or unclear settings. In studies that included confirmed cases the pooled sensitivity of chest CT was 93.1% (95%CI: 90.2 - 95.0 (65 studies, 5759 cases); and for X-ray 82.1% (95%CI: 62.5 to 92.7 (9 studies, 682 cases). Heterogeneity judged by visual assessment of the ROC plots was considerable. Two studies evaluated the diagnostic accuracy of point-of-care ultrasound and both reported zero false negatives (with 10 and 22 participants having undergone ultrasound, respectively). These studies only reported True Positive and False Negative data, therefore it was not possible to pool and derive estimates of specificity. In studies that included suspected cases, the pooled sensitivity of CT was 86.2% (95%CI: 71.9 to 93.8 (13 studies, 2346 participants) and specificity was 18.1% (95%CI: 3.71 to 55.8). Heterogeneity judged by visual assessment of the forest plots was high. Chest CT may give approximately the same proportion of positive results for patients with and without a SARS-CoV-2 infection: the chances of getting a positive CT result are 86% (95% CI: 72 to 94) in patient with a SARS-CoV-2 infection and 82% (95% CI: 44 to 96) in patients without. AUTHORS' CONCLUSIONS: The uncertainty resulting from the poor study quality and the heterogeneity of included studies limit our ability to confidently draw conclusions based on our results. Our findings indicate that chest CT is sensitive but not specific for the diagnosis of COVID-19 in suspected patients, meaning that CT may not be capable of differentiating SARS-CoV-2 infection from other causes of respiratory illness. This low specificity could also be the result of the poor sensitivity of the reference standard (RT-PCR), as CT could potentially be more sensitive than RT-PCR in some cases. Because of limited data, accuracy estimates of chest X-ray and ultrasound of the lungs for the diagnosis of COVID-19 should be carefully interpreted. Future diagnostic accuracy studies should avoid cases-only studies and pre-define positive imaging findings. Planned updates of this review will aim to: increase precision around the accuracy estimates for CT (ideally with low risk of bias studies); obtain further data to inform accuracy of chest X rays and ultrasound; and continue to search for studies that fulfil secondary objectives to inform the utility of imaging along different diagnostic pathways.

Citation bias in imaging research: are studies with higher diagnostic accuracy estimates cited more often?

OBJECTIVES: To assess the risk of citation bias in imaging diagnostic accuracy research by evaluating whether studies with higher accuracy estimates are cited more frequently than those with lower accuracy estimates. METHODS: We searched Medline for diagnostic accuracy meta-analyses published in imaging journals from January 2005 to April 2016. Primary studies from the meta-analyses were screened; those assessing the diagnostic accuracy of an imaging test and reporting sensitivity and specificity were eligible for inclusion. Studies not indexed in Web of Science, duplicates, and inaccessible articles were excluded. Topic (modality/subspecialty), study design, sample size, journal impact factor, publication date, times cited, sensitivity, and specificity were extracted for each study. Negative binomial regression was performed to evaluate the association of citation rate (times cited per month since publication) with Youden's index (sensitivity + specificity -1), highest sensitivity, and highest specificity, controlling for the potential confounding effects of modality, subspecialty, impact factor, study design, sample size, and source meta-analysis. RESULTS: There were 1016 primary studies included. A positive association between Youden's index and citation rate was present, with a regression coefficient of 0.33 (p = 0.016). The regression coefficient for sensitivity was 0.41 (p = 0.034), and for specificity, 0.32 (p = 0.15). CONCLUSION: A positive association exists between diagnostic accuracy estimates and citation rates, indicating that there is evidence of citation bias in imaging diagnostic accuracy literature. Overestimation of imaging test accuracy may contribute to patient harm from incorrect interpretation of test results. KEY POINTS: • Studies with higher accuracy estimates may be cited more frequently than those with lower accuracy estimates. • This citation bias could lead clinicians, reviews, and clinical practice guidelines to overestimate the accuracy of imaging tests, contributing to patient harm from incorrect interpretation of test results.

Selective Citation Practices in Imaging Research: Are Diagnostic Accuracy Studies With Positive Titles and Conclusions Cited More Often?

OBJECTIVE. The purpose of this study was to examine the existence of selective citation practices in the imaging literature by assessing whether diagnostic accuracy studies with positive titles or conclusions are cited more frequently than those with negative (or neutral) titles or conclusions. MATERIALS AND METHODS. MEDLINE was searched for meta-analyses of diagnostic accuracy studies published in imaging journals from January 2005 to April 2016. Primary studies from the meta-analyses were screened for eligibility. Titles and conclusions were classified independently in duplicate. A negative binomial regression analysis controlling for several confounding variables was performed to obtain regression coefficients; p values were obtained via likelihood ratio testing. RESULTS. A total of 995 primary studies were included. Fifty-one titles (5.1%) and 782 conclusions (78.6%) were positive or positive with qualifiers; 942 titles (94.7%) and 127 conclusions (12.8%) were neutral; and two titles (0.02%) and 86 conclusions (8.6%) were negative. Studies with positive, neutral, and negative titles were cited a mean of 0.66, 0.50, and 0.06 times per month. Studies with positive, neutral, and negative conclusions were cited a mean of 0.54, 0.42, and 0.34 times per month. Regression coefficients were 1.10 (95% CI, -0.08 to 2.20) and 0.91 (95% CI, -0.27 to 2.00) for positive and neutral titles, relative to negative titles. Regression coefficients were 0.19 (95% CI, 0.03-0.35) and 0.08 (95% CI, -0.12 to 0.27) for positive and neutral conclusions, relative to negative conclusions. Title and conclusion positivity demonstrated positive association with citation rate (p = 0.031 for both). CONCLUSION. Studies with positive titles or conclusions are cited more frequently in imaging diagnostic accuracy literature. This difference may contribute to overestimation of the accuracy of a test and, consequently, suboptimal patient outcomes.

Are Study and Journal Characteristics Reliable Indicators of "Truth" in Imaging Research?

Purpose To evaluate whether journal-level variables (impact factor, cited half-life, and Standards for Reporting of Diagnostic Accuracy Studies [STARD] endorsement) and study-level variables (citation rate, timing of publication, and order of publication) are associated with the distance between primary study results and summary estimates from meta-analyses. Materials and Methods MEDLINE was searched for meta-analyses of imaging diagnostic accuracy studies, published from January 2005 to April 2016. Data on journal-level and primary-study variables were extracted for each meta-analysis. Primary studies were dichotomized by variable as first versus subsequent publication, publication before versus after STARD introduction, STARD endorsement, or by median split. The mean absolute deviation of primary study estimates from the corresponding summary estimates for sensitivity and specificity was compared between groups. Means and confidence intervals were obtained by using bootstrap resampling; P values were calculated by using a t test. Results Ninety-eight meta-analyses summarizing 1458 primary studies met the inclusion criteria. There was substantial variability, but no significant differences, in deviations from the summary estimate between paired groups (P > .0041 in all comparisons). The largest difference found was in mean deviation for sensitivity, which was observed for publication timing, where studies published first on a topic demonstrated a mean deviation that was 2.5 percentage points smaller than subsequently published studies (P = .005). For journal-level factors, the greatest difference found (1.8 percentage points; P = .088) was in mean deviation for sensitivity in journals with impact factors above the median compared with those below the median. Conclusion Journal- and study-level variables considered important when evaluating diagnostic accuracy information to guide clinical decisions are not systematically associated with distance from the truth; critical appraisal of individual articles is recommended. © RSNA, 2017 Online supplemental material is available for this article.

Evaluation of adherence to STARD for abstracts in a diverse sample of diagnostic accuracy abstracts published in 2012 and 2019 reveals suboptimal reporting practices.

OBJECTIVES: To evaluate the completeness of reporting in a sample of abstracts on diagnostic accuracy studies before and after the release of Standards for Reporting of Diagnostic Accuracy Studies (STARD) for abstracts in 2017. METHODS: We included 278 diagnostic accuracy abstracts published in 2012 (N = 138) and 2019 (N = 140) and indexed in EMBASE. We analyzed their adherence to 10 items of the 11-item STARD for abstracts checklist, and we explored variability in reporting across abstract characteristics using multivariable Poisson modeling. RESULTS: Most of the 278 abstracts (75%) were published in discipline-specific journals, with a median impact factor of 2.9 (IQR: 1.9-3.7). The majority (41%) of abstracts reported on imaging tests. Overall, a mean of 5.4/10 (SD: 1.4) STARD for abstracts items was reported (range: 1.2-9.7). Items reported in less than one-third of abstracts included 'eligible patient demographics' (24%), 'setting of recruitment' (30%), 'method of enrollment' (18%), 'estimates of precision for accuracy measures' (26%), and 'protocol registration details' (4%). We observed substantial variability in reporting across several abstract characteristics, with higher adherence associated with the use of a structured abstract, no journal limit for abstract word count, abstract word count above the median, one-gate enrollment design, and prospective data collection. There was no evidence of increase in the number of reported items between 2012 and 2019 (5.2 vs 5.5 items; adjusted reporting ratio: 1.04 [95% CI: 0.98-1.10]). CONCLUSION: This sample of diagnostic accuracy abstracts revealed suboptimal reporting practices without improvement between 2012 and 2019. The test evaluation field could benefit from targeted knowledge translation strategies to improve completeness of reporting in abstracts.

Odor recognition memory as a function of odor-naming performance.

A series of experiments sought to clarify the relationship between odor naming and memory by manipulating odor label availability during a dual naming-memory task. Experiment 1 demonstrated that recognition memory and odor naming were both better when the naming task provided participants with odor label alternatives. Consistent and correct odor naming was associated with nearly perfect memory, whereas inconsistent or incorrect naming was associated with very weak memory if any at all. Experiment 2 showed that the availability of odor labels was effective at improving memory only if labels were available at both memory encoding and retrieval, suggesting that the labels were aiding memory by improving the identification of the odors. Odor naming was manipulated in Experiment 3 by varying the number of available labels from 4 to 16 during each odor-naming trial. As found in the previous experiments, naming and memory were strongly related in each of the labeling conditions. Experiment 4 showed that corrective naming feedback produced better memory performance but only when the feedback led to correct odor naming. It was concluded that perceptual processes related to matching olfactory input to acquired, multidimensional representations of odors play a critical role in both odor naming and episodic memory.

Reporting Guidelines for Imaging Research.

Suboptimal reporting in the publication of imaging research studies is a growing concern. Deficient and incomplete reporting prevents the evaluation of the validity, replicability, and the overall quality of the research. Reporting guidelines are checklists designed to guide researchers about the minimum information to be provided in research studies to allow for adequate quality appraisal and to assess generalizability. They are a powerful tool to allow key stakeholders such as journal editors, peer reviewers, funding agencies, and readers to better identify robust health research. The Enhancing the QUAlity and Transparency of Health Research Network is an international initiative that attempts to improve the reporting practices of a variety of health research study designs by providing the resources required to develop, disseminate, and implement reporting guidelines. In this review, we elaborate on the impact of good reporting on imaging research, and the different types of guidelines relevant for the various study designs applicable in imaging research.

Diagnostic accuracy of whole-brain computed tomography perfusion for detection of ischemic stroke in patients with mild neurological symptoms.

Mild and minor acute neurological symptoms may lead to diagnostic uncertainty, resulting in a heterogeneous group of patients with true ischemic events and stroke mimics with a potential for poor outcomes. More than half of ischemic stroke patients present as minor strokes (National Institutes of Health Stroke Scale score <6). Whole-brain computed tomography perfusion can be used as a diagnostic test for minor stroke, offering a potential method of reducing diagnostic uncertainty in these patients. We hypothesize that whole-brain computed tomography perfusion imaging features could accurately predict infarction in patients with minor neurological deficits. This retrospective chart review enrolled consecutive patients suspected of acute ischemic stroke with a National Institutes of Health Stroke Scale score <6, who underwent whole-brain computed tomography perfusion and follow-up diffusion-weighted magnetic resonance imaging at our institution. Sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios were calculated for whole-brain computed tomography perfusion, using follow-up diffusion-weighted magnetic resonance imaging as a reference standard. A total of 524 patients (mean age: 67 years; range: 17-96 years; 56% men) met the inclusion criteria. Patients were excluded for non-diagnostic ( n = 25) or missing maps ( n = 8) scans, non-ischemic findings ( n = 7), and lack of follow-up magnetic resonance imaging ( n = 336). The final analysis included 148 patients who underwent diffusion-weighted magnetic resonance imaging. Whole-brain computed tomography perfusion has a sensitivity of 0.57 (95% CI: 0.45-0.69) and a specificity of 0.82 (95% CI: 0.71-0.90). The positive and negative predictive values and positive and negative likelihood ratios were 75%, 67%, 3.09, and 0.53, respectively. Our analysis suggests that although whole-brain computed tomography perfusion may offer some value as an adjunctive test for improving confidence in offering stroke treatment, it is not sufficiently sensitive or specific to accurately predict cerebral infarcts in patients with minor neurological symptoms.

Characterization of the sniff magnitude test.

OBJECTIVE: To evaluate the potential utility of the Sniff Magnitude Test (SMT) as a clinical measure of olfactory function. DESIGN: Between-subject designs were used to compare the SMT and University of Pennsylvania Smell Identification Test (UPSIT) in study participants from a broad range of ages. SUBJECTS: A total of 361 individuals from retirement communities and an urban university and patients from an otolaryngology clinic. INTERVENTION: Study participants completed the SMT and UPSIT using standard procedures. MAIN OUTCOME MEASURES: The UPSIT was scored using standard procedures to calculate the number of correctly identified odors; a score that can range from 0 to 40 correct. The measure of olfactory function generated by the SMT is the "sniff magnitude ratio," defined as the mean sniff magnitude generated by the odor stimuli divided by the mean sniff magnitude to nonodorized air blanks. RESULTS: The SMT generally showed good agreement with UPSIT diagnostic categories, although SMT scores were only modestly elevated in the mild and modest hyposmia range of the UPSIT. Age-related decline in olfactory ability was evident on the UPSIT at younger ages than that seen with the SMT. As predicted, otolaryngology patients with olfactory complaints were found to be impaired on both the UPSIT and SMT. CONCLUSIONS: The SMT provides a novel method for evaluating the sense of smell that shows good general agreement with the UPSIT. Its minimal dependence on language and cognitive abilities provides some advantages over odor identification tests. There is some indication that the UPSIT may be more sensitive to olfactory (and/or nonolfactory) deficits. We conclude that sniffing behavior can be exploited for the clinical evaluation of olfaction. A comparison of performance on odor identification and sniffing tests may provide novel insight into the nature of olfactory problems in a variety of patient populations.