Radiographic parameter-driven decision tree reliably predicts aseptic mechanical failure of compressive osseointegration fixation.

Background and purpose - Compressive osseointegration fixation is an alternative to intramedullary fixation for endoprosthetic reconstruction. Mechanical failure of compressive osseointegration presents differently on radiographs than stemmed implants, therefore we aimed to develop a reliable radiographic method to determine stable integration.Patients and methods - 8 reviewers evaluated 11 radiographic parameters from 29 patients twice, 2 months apart. Interclass correlation coefficients (ICCs) were used to assess test-retest and inter-rater reliability. We constructed a fast and frugal decision tree using radiographic parameters with substantial test-retest agreement, and then tested using radiographs from a new cohort of 49 patients. The model's predictions were compared with clinical outcomes and a confusion matrix was generated.Results - 6 of 8 reviewers had non-significant intra-rater ICCs for ≥ one parameter; all inter-rater ICCs were highly reliable (p < 0.001). Change in length between the top of the spindle sleeve and bottom of the anchor plug (ICC 0.98), bone cortex hypertrophy (ICC 0.86), and bone pin hypertrophy (ICC 0.81) were used to create the decision tree. The sensitivity and specificity of the training cohort were 100% (95% CI 52-100) and 87% (CI 74-94) respectively. The decision tree demonstrated 100% (CI 40-100) sensitivity and 89% (CI 75-96) specificity with the test cohort.Interpretation - A stable spindle length and at least 3 cortices with bone hypertrophy at the implant interface predicts stable osseointegration; failure is predicted in the absence of bone hypertrophy at the implant interface if the pin sites show hypertrophy. Thus, our decision tree can guide clinicians as they follow patients with compressive osseo-integration implants.

Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study.

BACKGROUND: The Cochrane risk of bias tool is commonly criticized for having a low reliability. We aimed to investigate whether training of raters, with objective and standardized instructions on how to assess risk of bias, can improve the reliability of the Cochrane risk of bias tool. METHODS: In this pilot study, four raters inexperienced in risk of bias assessment were randomly allocated to minimal or intensive standardized training for risk of bias assessment of randomized trials of physical therapy treatments for patients with knee osteoarthritis pain. Two raters were experienced risk of bias assessors who served as reference. The primary outcome of our study was between-group reliability, defined as the agreement of the risk of bias assessments of inexperienced raters with the reference assessments of experienced raters. Consensus-based assessments were used for this purpose. The secondary outcome was within-group reliability, defined as the agreement of assessments within pairs of inexperienced raters. We calculated the chance-corrected weighted Kappa to quantify agreement within and between groups of raters for each of the domains of the risk of bias tool. RESULTS: A total of 56 trials were included in our analysis. The Kappa for the agreement of inexperienced raters with reference across items of the risk of bias tool ranged from 0.10 to 0.81 for the minimal training group and from 0.41 to 0.90 for the standardized training group. The Kappa values for the agreement within pairs of inexperienced raters across the items of the risk of bias tool ranged from 0 to 0.38 for the minimal training group and from 0.93 to 1 for the standardized training group. Between-group differences in Kappa for the agreement of inexperienced raters with reference always favored the standardized training group and was most pronounced for incomplete outcome data (difference in Kappa 0.52, p < 0.001) and allocation concealment (difference in Kappa 0.30, p = 0.004). CONCLUSIONS: Intensive, standardized training on risk of bias assessment may significantly improve the reliability of the Cochrane risk of bias tool.

Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol.

BACKGROUND: The Cochrane risk of bias (RoB) tool has been widely embraced by the systematic review community, but several studies have reported that its reliability is low. We aim to investigate whether training of raters, including objective and standardized instructions on how to assess risk of bias, can improve the reliability of this tool. We describe the methods that will be used in this investigation and present an intensive standardized training package for risk of bias assessment that could be used by contributors to the Cochrane Collaboration and other reviewers. METHODS/DESIGN: This is a pilot study. We will first perform a systematic literature review to identify randomized clinical trials (RCTs) that will be used for risk of bias assessment. Using the identified RCTs, we will then do a randomized experiment, where raters will be allocated to two different training schemes: minimal training and intensive standardized training. We will calculate the chance-corrected weighted Kappa with 95% confidence intervals to quantify within- and between-group Kappa agreement for each of the domains of the risk of bias tool. To calculate between-group Kappa agreement, we will use risk of bias assessments from pairs of raters after resolution of disagreements. Between-group Kappa agreement will quantify the agreement between the risk of bias assessment of raters in the training groups and the risk of bias assessment of experienced raters. To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals. DISCUSSION: This study will investigate whether the reliability of the risk of bias tool can be improved by training raters using standardized instructions for risk of bias assessment. One group of inexperienced raters will receive intensive training on risk of bias assessment and the other will receive minimal training. By including a control group with minimal training, we will attempt to mimic what many review authors commonly have to do, that is-conduct risk of bias assessment in RCTs without much formal training or standardized instructions. If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.