A Test Method for Identifying Selection Bias Risk in Prospective Controlled Clinical Therapy Trials Using the I2 Point Estimate.

OBJECTIVES: A test method is proposed for identifying potential selection bias risk in single prospective controlled clinical therapy trials that can be applied by trial reviewers. METHODS: The method is described in detail and was tested on eight randomised controlled trials (RCTs) with reported negative Berger-Exner test results as negative and on eight prospective, controlled cohort studies as positive controls. All 16 studies were identified by systematic literature search. RESULTS: The test method yielded negative results for all RCTs and positive results for six out of the eight cohort studies. CONCLUSION: All test results remained within the expected limits for both study types, suggesting a reasonably high accuracy for correctly identifying selection bias risk. However, the method does not provide the possibility to establish whether such bias risk has actually altered trial outcomes. Instead, a positive test result may provide an empirical basis for rating a trial as of high selection bias risk during trial appraisal.

Trial Number and Sample Size Do Not Affect the Accuracy of the I2-Point Estimate for Testing Selection Bias Risk in Meta-Analyses.

Aim This study aims to establish the test sensitivity and specificity of the I2-point estimate for testing selection bias in meta-analyses under the condition of large versus small trial sample size and large versus small trial number in meta-analyses and to test the null hypotheses that the differences are not statistically significant. Material and methods Simulation trials were generated in MS Excel (Microsoft Corp., Redmond, WA), each consisting of a sequence of subject ID (accession) numbers representing trial subjects, a random sequence of allocation to group A or B, and a random sequence of a simulated baseline variable ("age") per subject, ranging from 50 to 55. These simulation trials were included in five types of meta-analyses with large/small numbers of trials, as well as trials with large and small sample sizes. Half of the meta-analyses were artificially biased. All meta-analyses were tested using the I2-point estimate. The numbers of true positive (TP), false positive (FP), false negative (FN), and true negative (TN) test results were established. From these, the test sensitivity and specificity were computed for each of the meta-analysis types and compared. Results All non-biased meta-analyses yielded true negative, and all biased meta-analyses yielded true positive test results, regardless of trial number and trial sample size. No false positive or false negative test results were observed. Accordingly, test sensitivities and specificities of 100% for all meta-analysis types were established, and thus, both null hypotheses failed to be rejected. Conclusion The results suggest that trial number and sample size in a baseline variable meta-analysis do not affect the test accuracy of the I2-point estimate.

Application of the Composite Quality Score (CQS-2B) versus Cochrane's Risk of Bias tool (Version 2) in systematic reviews of clinical trials - an exploratory study.

Objectives: To explore whether systematic review conclusions generated from Cochrane's second version of its Risk of Bias tool (RoB 2) for trial appraisal differ when the Composite Quality Score, Version 2.B (CQS-2B) is used instead and to develop a testable hypothesis based on these findings. Methods: PubMed was searched for one single systematic review. From the review's accepted trials, data concerning effect estimates and overall bias risk according to the RoB 2 tool were extracted. All trial reports were appraised again using the CQS-2B. Datasets were stratified according to overall bias risk (RoB 2) or corroboration (C-) level (CQS-2B). The effect estimates from trials with 'low bias risk' (RoB 2) and with highest C-level (CQS-2B) were pooled separately. These pooled effect estimates were statistically and all clinical conclusions qualitatively compared. Results: The pooled effect estimates for trials with 'low bias risk' (RoB 2) were -0.07, 95% CI: -0.10 to -0.04 (I2 = 0.0%) and for the highest C-levels (CQS-2B) 0.08, 95% CI: -0.12 to -0.04 (I2 = 57.0%). The difference was statistically not significant (p = 0.70). Contrary to the RoB 2 tool, no clinical conclusions in line with the CQS-2B were made, because the effect estimates were judged to be erroneously overestimated, due to high risk of bias. Conclusion: A testable hypothesis was generated suggesting that trial appraisal using the CQS-2B may provide more conservative conclusions based on similar data than with the RoB 2 tool.

Inter-rater reliability of the extended Composite Quality Score (CQS-2).

Aim: To establish the inter-rater reliability of the Composite Quality Score (CQS-2) and to test the null hypothesis that it did not differ significantly from that of the first CQS version (CQS-1). Materials and methods: Four independent raters were selected to rate 45 clinical trial reports using CQS-1 and CQS-2. The raters remained unaware of each other's participation in this study until all rating had been completed. Each rater received only one rating template at a time in a random sequence for CQS-1 and CQS-2 rating. Raters completed each template and sent these back to the principal investigator. Each rater received their next template 2 weeks after submission of the completed previous template. The inter-rater reliabilities for the overall appraisal score of the CQS-1 and the CQS-2 were established by using the Brennan-Prediger coefficient (BPC). The coefficients of both CQS versions were compared by using the two-sample z-test. During secondary analysis, the BPCs for every criterion and each corroboration level for both CQS versions were established. Results: The BPC for the CQS-1 was 0.85 (95% CI: 0.64-1.00) and for the CQS-2 it was 1.00 (95% CI: 0.94-1.00), suggesting a very high inter-rater reliability for both. The difference between the two CQS versions was statistically not significant (p = 0.17). The null hypothesis was accepted. Conclusion: The CQS-2 is still under development, This study shows that it is associated with a very high inter-rater reliability, which did not statistically significantly differ from that of the CQS-1. The promising results of this study warrant further investigation in the applicability of the CQS-2 as an appraisal tool for prospective controlled clinical therapy trials.

Allocation concealment appraisal of clinical therapy trials using the extended Composite Quality Score (CQS-2)-An empirically based update.

Objectives: The objective of this study was to revise CQS-2/Criterion II concerning allocation concealment appraisal for prospective, controlled clinical therapy trials. Methods: Meta-analyses of trials with inadequate allocation concealment were tested for in-between trial heterogeneity (I2 > 0) due to imbalances in baseline variables. Meta-analyses with positive test results were used as a basis to deduce criteria for adequate allocation concealment. The CQS-2/Criterion II was reformulated in line with the findings. Result: One suitable meta-analysis was identified. Two forest plots with data from five and four trials with inadequate/unclear allocation concealment were selected for testing. In addition, a total of five trials with adequate allocation concealment were identified. The meta-analysis test results were positive, and keywords for the judgment of adequate allocation concealment were extracted verbatim from the text of the meta-analysis. The extracted keywords indicated "central allocation" as the main criterion for adequate allocation concealment. Criterion II of the CQS-2 was revised accordingly. Conclusion: Criterion II of the CQS-2 trial appraisal tool was revised. The revised appraisal tool was specified as version CQS-2B.

The composite quality score for the appraisal of prospective controlled clinical therapy trials in systematic reviews and its limits.

Systematic reviews of prospective controlled clinical therapy trials are one of the most important sources of information in modern medicine. Besides the systematic search for and statistical pooling of current clinical trial data for a particular type of therapy, systematic reviews also have the task of appraising the quality of trial results. The quality of trial results may be diminished by low internal trial validity, due to systematic error (bias). A high risk of bias may likely cause the reported trial results to be diverted from the actual true therapeutic effect and thus render it unsuitable for clinical guidance. According to the Cochrane Collaboration, the risk of bias in clinical therapy trials should be assessed using its Risk of Bias tool, Version 2 (RoB 2). However, the tool has been established to have poor inter-rater reliability, with a limited empirical evidence base and described as complex and demanding. Against this background, the composite quality score (CQS) has been developed as a possible alternative trial appraisal tool, characterised by high epistemic rigour, empirical evidence base, inter-rater reliability and ease of use. This article presents the current evidence of the CQS and its limitations.

The Composite Quality Score (CQS) as an Appraisal Tool for Prospective, Controlled Clinical Therapy Trials: Rationale and Current Evidence.

BACKGROUND: Current evidence appraisal concepts, such as the Assessment, Development and Evaluation (GRADE) approach and Cochrane's Risk of Bias (RoB) tool, rely on assumptions related to the classic problem of inductive reasoning and may suffer from insufficient inter-rater reliability. DISCUSSION: The Composite Quality Score (CQS) has emerged as a possible trial appraisal tool that does not rely on inductive assumptions and has been shown to be of potentially very high inter-rater reliability. CONCLUSION: Although the current CQS concept is still under development, its current evidence is encouraging and justifies further study. This article presents the rationale and currently available research concerning the CQS and shows where further research is required.

Extension of the Composite Quality Score (CQS) as an appraisal tool for prospective, controlled clinical therapy trials-A systematic review of meta-epidemiological evidence.

AIM: To conduct a survey of current meta-epidemiological studies to identify additional trial design characteristics that may be associated with significant over- or underestimation of the treatment effect and to use such identified characteristics as a basis for the formulation of new CQS appraisal criteria. MATERIALS AND METHODS: We retrieved eligible studies from two systematic reviews on this topic (latest search May 2015) and searched the databases PubMed and Embase for further studies from June 2015 -March 2022. All data were extracted by one author and verified by another. Sufficiently homogeneous estimates from single studies were pooled using random-effects meta-analysis. Trial design characteristics associated with statistically significant estimates from single datasets (which could not be pooled) and meta-analyses were used as a basis to formulate new or amend existing CQS criteria. RESULTS: A total of 38 meta-epidemiological studies were identified. From these, seven trial design characteristics associated with statistically significant over- or underestimation of the true therapeutic effect were found. CONCLUSION: One new criterion concerning double-blinding was added to the CQS, and the original criteria for concealing the random allocation sequence and for minimum sample size were amended.

Risk of selection bias assessment in the NINDS rt-PA stroke study.

OBJECTIVES: The NINDS rt-PA Stroke Study is frequently cited in support of alteplase for acute ischemic stroke within 3 h of symptom onset. Multiple post-hoc reanalyses of this trial have been published to adjust for a baseline imbalance in stroke severity. We performed a risk of selection bias assessment and reanalyzed trial data to determine if the etiology of this baseline imbalance was more likely due to random chance or randomization errors. METHODS: A risk of selection bias assessment was conducted using signaling questions from the Cochrane Risk of Bias 2 (ROB 2) tool. Four sensitivity analyses were conducted on the trial data based on the randomization process: assessment of imbalances in allocation in unique strata; adherence to a pre-specified restriction on randomization between time strata at each randomization center; assessment of differences in baseline computed tomography (CT) results in unique strata; and comparison of baseline characteristics between allocation groups within each time strata. A multivariable logistic regression model was used to compare reported treatment effects with revised treatment effects after adjustment of baseline imbalances identified in the sensitivity analyses. RESULTS: Based on criteria from the ROB 2 tool, the risk of bias arising from the randomization process was high. Sensitivity analyses found 11 of 16 unique strata deviated from the expected 1:1 allocation ratio. Three randomization centers violated an apriori rule regarding a maximum difference in allocation between the time strata. Three unique strata had imbalances in baseline CT results that prognostically favored alteplase. Four imbalances in baseline characteristics were identified in the 91-180-min time stratum that all prognostically favored alteplase and were consistent with a larger alteplase treatment effect size compared to the 0-90-min time stratum. After adjustments for baseline imbalances, all reported treatment effects were reduced. Three out of seven originally positive reported results were revised to non-significant. CONCLUSION: This risk of selection bias assessment revealed a high risk of selection bias in the NINDS rt-PA Stroke Study. Sensitivity analyses conducted based on the randomization process supported this assessment. Baseline imbalances in the trial were more likely due to randomization errors than random chance. Adjusted analyses accounting for baseline imbalances revealed a reduction in reported treatment effects supporting the presence of selection bias in the trial. Treatment decisions and guideline recommendations based on the original treatment effect reported in the NINDS rt-PA Stroke Study should be done cautiously.

The Composite Quality Score (CQS) as a trial appraisal tool: inter-rater reliability and rating time.

OBJECTIVE: To establish the CQS inter-rater reliability and rating time and to compare both against that of the Jadad scale and Cochrane's Risk of Bias Tool (ROBT). MATERIAL AND METHODS: Four independent raters rated 45 trial reports. The inter-rater reliability was established by use of the Brennan-Prediger coefficient (BPC). The coefficients were compared using the two-sample z-test. Secondary analysis included comparison of the inter-rater reliability of the randomization component of all tools, as well as of the allocation concealment component of the CQS to that of the ROBT. The mean rating time with standard deviation (SD) for each tool was determined using one-way repeated measures analysis of variance. Post hoc comparisons were made using the Tukey-Kramer adjustment for three pair-wise multiple comparisons. RESULTS: The inter-rater reliability was significantly higher for the CQS (BPC, 95% CI: 0.95, 0.87-1.00) compared to Jadad (0.70, 0.58-0.82) (adjusted p = 0.0005) and most components of ROBT. The mean (SD) time to complete the CQS (4.0 (1.0) min) did not differ significantly from that of the Jadad scale 4.8 (1.1) min (adjusted p = 0.11), but was significantly shorter compared to that of the ROBT 15.3 (5.9) min (adjusted p < 0.0001). CONCLUSIONS: The results suggest the CQS to be a very reliable and fast trial appraisal tool. Clinical relevance The higher the inter-rater reliability, the higher the probability that trial results reflect therapeutic truth. The CQS will need to take further bias sources into consideration, in order to increase its utility.

Are Most of the Published Clinical Trial Results in Restorative Dentistry Invalid? An Empirical Investigation.

BACKGROUND: To establish the number of invalid clinical trial reports in restorative dentistry, due to lack of effective randomisation and/or inadequate sample size and whether this number changed, during the 1990-2019 period. METHODS: Databases were searched up to 14 July 2019 without limitations regarding publication language. A Journal hand search and reference check were conducted for trial reports. Selection criteria were: reporting on a prospective, controlled clinical trial; relevance to placing direct tooth restorations in human vital teeth; direct comparison between restorative materials concerning tooth restoration longevity; trial report published from 1990. Randomisation reported (Yes/No) and treatment group sample size ≥ 200 were applied as criteria, using the deductive falsification approach for trial report appraisal. RESULTS: 683 trial reports were appraised. 660 lacked effective randomisation. Of the remaining 23 reports, only 2 included a sample size of more than 200 restored teeth (mean number per treatment group 87; Standard deviation = 108.51). 92.5% of all treatment groups had a sample size of < 200. Randomisation reporting increased and sample size remained essentially unchanged between 1990 and 2019. CONCLUSION: Most of the published clinical trial results in restorative dentistry were judged invalid, due to lack of effective randomisation and adequate sample size. These results are in line with previous findings. Evidence-based recommendations on how to improve trial methodology are available in the dental/medical literature.

Is the Deductive Falsification Approach a Better Basis for Clinical Trial Appraisal?

BACKGROUND: Inductive reasoning relies on an infinite regress without sufficient factual basis and verification is at any time vulnerable to single contrary observation. Thus, appraisal based on inductive verification, as applied in current clinical trial appraisal scales, checklists or grading systems, cannot prove or justify trial validity. DISCUSSION: Trial appraisal based on deductive falsification can identify invalid trials and give evidence for the recommendation to exclude these from clinical decision-making. Such appraisal remains agnostic towards corroborated trials that pass all appraisal criteria. The results of corroborated trials cannot be considered more robust than falsified trials since nothing within a particular set of complied trial criteria can give certainty for trial compliance with any other appraisal criterion in future. A corroborated trial may or may not reflect therapeutic truth and may thus be the basis for clinical guidance, pending results of any future trial re-appraisal. CONCLUSION: Trial grading following appraisal based on deductive falsification should be binary (0 = Invalid or 1 = Unclear) and single component scores should be multiplied. Appraisal criteria for the judgment of trial characteristics require a clear rationale, quantification of such rationale and empirical evidence concerning the effect of trial characteristics on trial results.

Reports of uncontrolled clinical trials for directly placed restorations in vital teeth.

Uncontrolled trials are criticized as unreliable. This study aimed to establish how the number of published reports from uncontrolled clinical trials compares to that of controlled trials for directly placed restorations in vital teeth and whether their annual number is increasing, stable or decreasing. PubMed was searched and suitable citations of uncontrolled and controlled trial reports published between 1990-2016 were included. Reference check and hand searching were conducted. The median annual report number with 25 and 75% percentile was calculated for both types of trials. 695 reports were found. The median number of reports per year was 4 (3-7) and 22 (15-26) from uncontrolled and controlled trials, respectively. A statistically significant decreasing ratio of uncontrolled to controlled trial reports was observed (p = 0.01) by linear regression analysis. The number of reports of uncontrolled clinical trials listed in PubMed over the last 27 years appears at least five times smaller than that of controlled clinical trials and its number in relation to that of controlled trials seems to decrease over time.

The logic behind the use of fissure sealant retention as a proxy outcome measure for dental caries prevention.

Fissure sealant retention is traditionally considered as a proxy measure for caries prevention. This study investigated the logic behind this proposition, and its validity. A logical framework of the proposition was established. The mechanism of caries development was transferred into a Directed acyclic graph, and this was used to investigate the logical framework. The sensitivity and specificity of full sealant retention in the prediction of dental carious lesion development and the number of false positive/false negative prediction rates were computed. The sensitivity/specificity was statistically compared to that of random values. A contradiction in the logical framework was identified. The mean sensitivity/specificity was 37.9% (SD = 27.8%) and 67.6% (SD = 28.4%), respectively. When these values were compared against random values (30.5%, SD = 25.7% and 58.7%, SD = 31.6%), a non-significant sensitivity (P = 0.06) and a borderline higher specificity (P = 0.04) were observed. The overall false prediction rate was 33.7%, with 16.9% and 16.8% false negative and false positive predictions, respectively. The sensitivity/specificity was too low and the false prediction rate was too high to consider retention a valid proxy for caries prevention. The logic behind the investigated proposition is flawed, contradicted by the current empirical evidence, and thus invalid.

Reports of uncontrolled clinical trials for directly placed restorations in vital teeth

Abstract Uncontrolled trials are criticized as unreliable. This study aimed to establish how the number of published reports from uncontrolled clinical trials compares to that of controlled trials for directly placed restorations in vital teeth and whether their annual number is increasing, stable or decreasing. PubMed was searched and suitable citations of uncontrolled and controlled trial reports published between 1990-2016 were included. Reference check and hand searching were conducted. The median annual report number with 25 and 75% percentile was calculated for both types of trials. 695 reports were found. The median number of reports per year was 4 (3-7) and 22 (15-26) from uncontrolled and controlled trials, respectively. A statistically significant decreasing ratio of uncontrolled to controlled trial reports was observed (p = 0.01) by linear regression analysis. The number of reports of uncontrolled clinical trials listed in PubMed over the last 27 years appears at least five times smaller than that of controlled clinical trials and its number in relation to that of controlled trials seems to decrease over time.

Caries-Preventive Effect of High-Viscosity Glass Ionomer and Resin-Based Fissure Sealants on Permanent Teeth: A Systematic Review of Clinical Trials.

BACKGROUND: Glass-ionomers are traditionally regarded to be inferior to resin as fissure sealants in protecting teeth from dental caries, due to their comparatively lower retention rate. Unlike low-viscosity glass-ionomers, high-viscosity glass-ionomer cements (HVGIC) are placed as sealants by pressing the material into pits and fissures with a petroleum-jelly-coated index finger. Hence, HVGIC sealants are assumed to penetrate pits and fissures deeper, resulting in a higher material retention rate, which may increase its caries-preventive effect. METHODS: The aim of this review was to answer the question as to whether, in patients with fully erupted permanent molar teeth, HVGIC based fissure sealants are less effective to protect against dental carious lesions in occlusal pits and fissures than resin-based fissure sealants? A systematic literature search in eight databases was conducted. Heterogeneity of accepted trials and imprecision of the established evidence were assessed. Extracted sufficiently homogenous datasets were pooled by use of a random-effects meta-analysis. Internal trial validity was evaluated. The protocol of this systematic review was registered with the International Prospective Register of Systematic Reviews (PROSPERO / Nr.: CRD42015016007). RESULTS: Seven clinical trials were provisionally included for further review. Of these, one was excluded. Seven trial reports reporting on six trials were accepted. From these, 11 datasets were extracted and pooled in four meta-analyses. The results suggest no statistically significant differences after up to 48 months and borderline significant differences in favour of HVGIC sealants after 60 months (RR 0.29; 95% CI: 0.09-0.95; p = 0.04 / RD -0.07; 95% CI: -0.14, -0.01). The point estimates and upper confidence levels after 24, 36, 48 and 60 months of RR 1.36; RR 0.90; RR 0.62; RR 0.29 and 2.78; 1.67; 1.21; 0.95, respectively, further suggest a chronological trend in favour of HVGIC above resin-based sealants. The internal trial validity was judged to be low and the bias risk high for all trials. Imprecision of results was considered too high for clinical guidance. CONCLUSION: It can be concluded that: (i) Inferiority claims against HVGIC in comparison to resin-based sealants as current gold-standard are not supported by the clinical evidence; (ii) The clinical evidence suggests similar caries-preventive efficacy of HVGIC and resin-based sealants after a period of 48 months in permanent molar teeth but remains challenged by high bias risk; (iii) Evidence concerning a possible superiority of HVGIC above resin-based sealants after 60 months is poor (even if the high bias risk is disregarded) due to imprecision and requires corroboration through future research.

Are high-viscosity glass-ionomer cements inferior to silver amalgam as restorative materials for permanent posterior teeth? A Bayesian analysis.

BACKGROUND: To develop a synthesis within a Bayesian probability framework of previously established evidence, in order to derive an overall conclusion about the hypothesis (H1): 'High-viscosity glass-ionomer cements (HVGIC) are inferior to silver amalgam as (load bearing) restorative materials for permanent posterior teeth'. METHODS: Following Bayesian method, the prior Odds that H1 is true (established from past uncontrolled clinical longitudinal and laboratory trials), the Likelihood Ratio incorporating new evidence (established from recent meta-epidemiological studies and systematic reviews of controlled clinical trials), as well as the posterior hypothesis Odds in view of the new evidence, were calculated. RESULTS: The prior Odds that HVGICs are clinically inferior to amalgam as restorative materials in posterior permanent teeth in relation to the hypothesis that this is not so was 1.12 to 1. The Likelihood Ratio based on new evidence in favor the hypothesis was zero and the subsequent posterior Odds 0 to 1. Therefore, based on the new evidence, the Odds that HVGICs are clinically inferior to amalgam as restorative materials in posterior permanent teeth degreased from 1.12 to zero. CONCLUSION: The current evidence suggests lack of support for the hypothesis that high-viscosity glass-ionomer cements are inferior to silver amalgam as restorative materials for permanent posterior teeth. Should future research to this topic uphold the current findings, a wider range of clinical benefits for both patient and care provider, beyond appropriate restoration longevity for placing HVGIC based restorations may apply.

Do Laboratory Results Concerning High-Viscosity Glass-Ionomers versus Amalgam for Tooth Restorations Indicate Similar Effect Direction and Magnitude than that of Controlled Clinical Trials? - A Meta-Epidemiological Study.

BACKGROUND: A large percentage of evidence concerning dental interventions is based on laboratory research. The apparent wealth of laboratory evidence is sometimes used as basis for clinical inference and recommendations for daily dental practice. In this study two null-hypotheses are tested: whether trial results from laboratory and controlled clinical trials concerning the comparison of high-viscosity glass-ionomer cements (HVGIC) to amalgam for restorations placed in permanent posterior teeth have: (i) similar effect direction and (ii) similar effect magnitude. METHODS: 7 electronic databases were searched, as well as reference lists. Odds ratios (OR) and Standardised Mean Differences (SMD) with 95% Confidence intervals were computed for extracted dichotomous and continuous data, respectively. Pooled effect estimates for laboratory and clinical data were computed to test for effect direction. Odds ratios were converted into SMDs. SMDs from laboratory and clinical data were statistically compared to test for differences in effect magnitude. The analysed results were further investigated within the context of potential influencing or confounding factors using a Directed acyclic graph. RESULTS: Of the accepted eight laboratory and nine clinical trials, 13 and 21 datasets could be extracted, respectively. The pooled results of the laboratory datasets were highly statistically significant in favor of amalgam. No statistically significant differences, between HVGICs and amalgam, were identified for clinical data. For effect magnitude, statistically significant differences between clinical and laboratory trial results were found. Both null-hypotheses were rejected. CONCLUSION: Laboratory results concerning high-viscosity glass-ionomers versus amalgam for tooth restorations do not indicate similar effect direction and magnitude than that of controlled clinical trials.