Replicability of simulation studies for the investigation of statistical methods: the RepliSims project.

Results of simulation studies evaluating the performance of statistical methods can have a major impact on the way empirical research is implemented. However, so far there is limited evidence of the replicability of simulation studies. Eight highly cited statistical simulation studies were selected, and their replicability was assessed by teams of replicators with formal training in quantitative methodology. The teams used information in the original publications to write simulation code with the aim of replicating the results. The primary outcome was to determine the feasibility of replicability based on reported information in the original publications and supplementary materials. Replicasility varied greatly: some original studies provided detailed information leading to almost perfect replication of results, whereas other studies did not provide enough information to implement any of the reported simulations. Factors facilitating replication included availability of code, detailed reporting or visualization of data-generating procedures and methods, and replicator expertise. Replicability of statistical simulation studies was mainly impeded by lack of information and sustainability of information sources. We encourage researchers publishing simulation studies to transparently report all relevant implementation details either in the research paper itself or in easily accessible supplementary material and to make their simulation code publicly available using permanent links.

Perspective; recommendations for improved patient participation in decision-making for geriatric patients in acute surgical settings.

Geriatric patients often present to the hospital in acute surgical settings. In these settings, shared decision-making as equal partners can be challenging. Surgeons should recognize that geriatric patients, and frail patients in particular, may sometimes benefit from de-escalation of care in a palliative setting rather than curative treatment. To provide more person-centred care, better strategies for improved shared decision-making need to be developed and implemented in clinical practice. A shift in thinking from a disease-oriented paradigm to a patient-goal-oriented paradigm is required to provide better person-centred care for older patients. We may greatly improve the collaboration with patients if we move parts of the decision-making process to the pre-acute phase. In the pre-acute phase appointing legal representatives, having goals of care conversations, and advance care planning can help give physicians an idea of what is important to the patient in acute settings. When making decisions as equal partners is not possible, a greater degree of physician responsibility may be appropriate. Physicians should tailor the "sharedness" of the decision-making process to the needs of the patient and their family.

Effects of reduction technique for acute anterior shoulder dislocation without sedation or intra-articular pain management: a systematic review and meta-analysis.

INTRODUCTION: Anterior shoulder dislocations are commonly seen in the emergency department for which several closed reduction techniques exist. The aim of this systematic review is to identify the most successful principle of closed reduction techniques for an acute anterior shoulder dislocation in the emergency department without the use of sedation or intra-articular lidocaine injection. METHODS: A literature search was conducted up to 15-08-2022 in the electronic databases of PubMed, Embase and CENTRAL for randomized and observational studies comparing two or more closed reduction techniques for anterior shoulder dislocations. Included techniques were grouped based on their main operating mechanism resulting in a traction-countertraction (TCT), leverage and biomechanical reduction technique (BRT) group. The primary outcome was success rate and secondary outcomes were reduction time and endured pain scores. Meta-analyses were conducted between reduction groups and for the primary outcome a network meta-analysis was performed. RESULTS: A total of 3118 articles were screened on title and abstract, of which 9 were included, with a total of 987 patients. Success rates were 0.80 (95% CI 0.74; 0.85), 0.81 (95% CI 0.63; 0.92) and 0.80 (95% CI 0.56; 0.93) for BRT, leverage and TCT, respectively. No differences in success rates were observed between the three separate reduction groups. In the network meta-analysis, similar yet more precise effect estimates were found. However, in a post hoc analysis the BRT group was more successful than the combined leverage and TCT group with a relative risk of 1.33 (95% CI 1.19, 1.48). CONCLUSION: All included techniques showed good results with regard to success of reduction. The BRT might be the preferred technique for the reduction of an anterior shoulder dislocation, as patients experience the least pain and it results in the fastest reduction.

A recalibrated prediction model can identify level-1 trauma patients at risk of nosocomial pneumonia.

INTRODUCTION: Nosocomial pneumonia has poor prognosis in hospitalized trauma patients. Croce et al. published a model to predict post-traumatic ventilator-associated pneumonia, which achieved high discrimination and reasonable sensitivity. We aimed to externally validate Croce's model to predict nosocomial pneumonia in patients admitted to a Dutch level-1 trauma center. MATERIALS AND METHODS: This retrospective study included all trauma patients (≥ 16y) admitted for > 24 h to our level-1 trauma center in 2017. Exclusion criteria were pneumonia or antibiotic treatment upon hospital admission, treatment elsewhere > 24 h, or death < 48 h. Croce's model used eight clinical variables-on trauma severity and treatment, available in the emergency department-to predict nosocomial pneumonia risk. The model's predictive performance was assessed through discrimination and calibration before and after re-estimating the model's coefficients. In sensitivity analysis, the model was updated using Ridge regression. RESULTS: 809 Patients were included (median age 51y, 67% male, 97% blunt trauma), of whom 86 (11%) developed nosocomial pneumonia. Pneumonia patients were older, more severely injured, and underwent more emergent interventions. Croce's model showed good discrimination (AUC 0.83, 95% CI 0.79-0.87), yet predicted probabilities were too low (mean predicted risk 6.4%), and calibration was suboptimal (calibration slope 0.63). After full model recalibration, discrimination (AUC 0.84, 95% CI 0.80-0.88) and calibration improved. Adding age to the model increased the AUC to 0.87 (95% CI 0.84-0.91). Prediction parameters were similar after the models were updated using Ridge regression. CONCLUSION: The externally validated and intercept-recalibrated models show good discrimination and have the potential to predict nosocomial pneumonia. At this time, clinicians could apply these models to identify high-risk patients, increase patient monitoring, and initiate preventative measures. Recalibration of Croce's model improved the predictive performance (discrimination and calibration). The recalibrated model provides a further basis for nosocomial pneumonia prediction in level-1 trauma patients. Several models are accessible via an online tool. LEVEL OF EVIDENCE: Level III, Prognostic/Epidemiological Study.

Concepts, utilization, and perspectives on the Dutch Nationwide Trauma registry: a position paper.

Over the last decades, the Dutch trauma care have seen major improvements. To assess the performance of the Dutch trauma system, in 2007, the Dutch Nationwide Trauma Registry (DNTR) was established, which developed into rich source of information for quality assessment, quality improvement of the trauma system, and for research purposes. The DNTR is one of the most comprehensive trauma registries in the world as it includes 100% of all trauma patients admitted to the hospital through the emergency department. This inclusive trauma registry has shown its benefit over less inclusive systems; however, it comes with a high workload for high-quality data collection and thus more expenses. The comprehensive prospectively collected data in the DNTR allows multiple types of studies to be performed. Recent changes in legislation allow the DNTR to include the citizen service numbers, which enables new possibilities and eases patient follow-up. However, in order to maximally exploit the possibilities of the DNTR, further development is required, for example, regarding data quality improvement and routine incorporation of health-related quality of life questionnaires. This would improve the quality assessment and scientific output from the DNTR. Finally, the DNTR and all other (European) trauma registries should strive to ensure that the trauma registries are eligible for comparisons between countries and healthcare systems, with the goal to improve trauma patient care worldwide.

Publication rates in small German trials remained low five years after trial completion.

OBJECTIVE: To investigate publication rates in small trials and to explore which factors are associated with publication rates in small trials, including sample size, the type and number of primary and secondary outcomes. STUDY DESIGN AND SETTING: We studied a subgroup of 'small' trials from a pre-existing dataset (IntoValue), containing German trials completed between 2009 and 2017. Small trials were defined as phase II-III, III and IV trials with 150 or fewer participants. We performed an updated publication search and collected additional data from online trial records. RESULTS: Out of 499 trials, 325 (65%) trials published their results in a journal article or dissertation. Median time-to-publication was 3.41 years (95% CI: 3.04-4.10). Planned sample size was not associated with publication rates, but the difference between planned and achieved sample size was (per 10% unsuccessfully recruited participants, HR = 0.95, 95% CI: 0.91-1.00). Phase III vs. II-III trials, studied intervention (device vs. other) and clearly vs. unclearly defined primary outcomes predicted a higher likelihood of earlier publication. CONCLUSION: About 35% of small trials in Germany remain unpublished, even after an extensive follow-up period of over 9 years. Publication rates are low and were associated with sample size, trial phase and type of intervention.

Real-World Metastatic Renal Cell Carcinoma Treatment Patterns and Clinical Outcomes in The Netherlands.

The number of treatment options for patients with metastatic renal cell carcinoma (mRCC) has significantly grown in the last 15 years. Although randomized controlled trials are fundamental in investigating mRCC treatment efficacy, their external validity can be limited. Therefore, the efficacy of the different treatment options should also be evaluated in clinical practice. We performed a chart review of electronic health records using text mining software to study the current treatment patterns and outcomes. mRCC patients from two large hospitals in the Netherlands, starting treatment between January 2015 and May 2020, were included. Data were collected from electronic health records using a validated text mining tool. Primary endpoints were progression-free survival (PFS) and overall survival (OS). Statistical analyses were performed using the Kaplan-Meier method. Most frequent first-line treatments were pazopanib (n = 70), sunitinib (n = 34), and nivolumab with ipilimumab (n = 28). The overall median PFS values for first-line treatment were 15.7 months (95% confidence interval [95%CI], 8.8-20.7), 16.3 months (95%CI, 9.3-not estimable [NE]) for pazopanib, and 6.9 months (95% CI, 4.4-NE) for sunitinib. The overall median OS values were 33.4 months (95%CI, 28.1-50.9 months), 39.3 months (95%CI, 29.5-NE) for pazopanib, and 28.1 months (95%CI, 7.0-NE) for sunitinib. For nivolumab with ipilimumab, median PFS and median OS were not reached. Of the patients who finished first- and second-line treatments, 64 and 62% received follow-up treatments, respectively. With most patients starting on pazopanib and sunitinib, these real-world treatment outcomes were most likely better than in pivotal trials, which may be due to extensive follow-up treatments.

External validation of the U-HIP prediction model for in-hospital mortality in geriatric hip fracture patients.

INTRODUCTION: Identification of high-risk hip fracture patients in an early stage is vital for guiding surgical management and shared decision making. To objective of this study was to perform an external international validation study of the U-HIP prediction model for in-hospital mortality in geriatric patients with a hip fracture undergoing surgery. MATERIALS AND METHODS: In this retrospective cohort study, data were used from The American College of Surgeons National Surgical Quality Improvement Program. Patients aged 70 years or above undergoing hip fracture surgery were included. The discrimination (c-statistic) and calibration of the model were investigated. RESULTS: A total of 25,502 patients were included, of whom 618 (2.4%) died. The mean predicted probability of in-hospital mortality was 3.9% (range 0%-55%). The c-statistic of the model was 0.74 (95% CI 0.72-0.76), which was comparable to the c-statistic of 0.78 (95% CI 0.71-0.85) that was found in the development cohort. The calibration plot indicated that the model was slightly overfitted, with a calibration-in-the-large of 0.015 and a calibration slope of 0.780. Within the subgroup of patients aged between 70 and 85, however, the c-statistic was 0.78 (95% CI 0.75-0.81), with good calibration (calibration slope 0.934). DISCUSSION AND CONCLUSION: The U-HIP model for in-hospital mortality in geriatric hip fractures was externally validated in a large international cohort, and showed a good discrimination and fair calibration. This model is freely available online and can be used to predict the risk of mortality, identify high-risk patients and aid clinical decision making.

[Trials, observational research and the real world]. / Trials, observationeel onderzoek en de 'echte' wereld.

The treatment effect found in a randomized trial does not always correspond to the effect of the treatment in daily practice. To estimate the applicability of the results of a trial, a comparison can be made with the results of observational research. In this commentary we discuss such a comparison between the results of the TIME trial and the analysis of the observational DUCA database. Both compared open and minimally invasive oesophageal resection, yet results were strikingly different. We discuss nine possible explanations for the differences found in the effects of the two treatments.

Effectiveness and toxicity of lenvatinib in refractory thyroid cancer: Dutch real-life data.

OBJECTIVE: The SELECT trial showed progression-free survival (PFS) benefit for lenvatinib for advanced radioiodine-refractory differentiated thyroid cancer (RAI-refractory or RR-DTC) patients, on which current clinical practice is based. We assessed whether the effectiveness and toxicity of lenvatinib in real-life clinical practice in the Netherlands were comparable to the pivotal SELECT trial. METHODS: From three Dutch centres Electronic Health Records (EHRs) of patients treated in the lenvatinib compassionate use program or as standard of care were reviewed and checked for SELECT eligibility criteria. Baseline characteristics, safety, and efficacy measures were compared and PFS and overall survival (OS) were calculated. Furthermore, PFS was compared to estimates of PFS reported in other studies. RESULTS: A total of 39 DTC patients with a median age of 62 years were analysed. Of these, 27 patients (69%) did not fulfil the SELECT eligibility criteria. The most common grade ≥3 toxicities were hypertension (n = 11, 28%), diarrhoea (n = 7, 18%), vomiting (n = 4, 10%), and gallbladder disease (n = 3, 8%). Median PFS and median OS were 9.7 (95% confidence interval (CI): 4.0-15.5) and 18.3 (95% CI: 4.9-31.7) months, respectively, response rate was 38% (95% CI: 23-54%). PFS in the Dutch real-life situation was comparable to previous real-life studies, but inferior to PFS as shown in the SELECT trial (P = 0.04). CONCLUSIONS: PFS in our non-trial population was significantly shorter than in the SELECT trial population. In the interpretation of results, differences in the real-life population and the SELECT study population regarding patient characteristics should be taken into account.

Measurement error in continuous endpoints in randomised trials: Problems and solutions.

In randomised trials, continuous endpoints are often measured with some degree of error. This study explores the impact of ignoring measurement error and proposes methods to improve statistical inference in the presence of measurement error. Three main types of measurement error in continuous endpoints are considered: classical, systematic, and differential. For each measurement error type, a corrected effect estimator is proposed. The corrected estimators and several methods for confidence interval estimation are tested in a simulation study. These methods combine information about error-prone and error-free measurements of the endpoint in individuals not included in the trial (external calibration sample). We show that, if measurement error in continuous endpoints is ignored, the treatment effect estimator is unbiased when measurement error is classical, while Type-II error is increased at a given sample size. Conversely, the estimator can be substantially biased when measurement error is systematic or differential. In those cases, bias can largely be prevented and inferences improved upon using information from an external calibration sample, of which the required sample size increases as the strength of the association between the error-prone and error-free endpoint decreases. Measurement error correction using already a small (external) calibration sample is shown to improve inferences and should be considered in trials with error-prone endpoints. Implementation of the proposed correction methods is accommodated by a new software package for R.

Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective.

It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out-of-sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between derivation and validation data is common, the impact on the out-of-sample performance is not well studied. Using analytical and simulation approaches, we examined out-of-sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.

How variation in predictor measurement affects the discriminative ability and transportability of a prediction model.

BACKGROUND AND OBJECTIVE: Diagnostic and prognostic prediction models often perform poorly when externally validated. We investigate how differences in the measurement of predictors across settings affect the discriminative power and transportability of a prediction model. METHODS: Differences in predictor measurement between data sets can be described formally using a measurement error taxonomy. Using this taxonomy, we derive an expression relating variation in the measurement of a continuous predictor to the area under the receiver operating characteristic curve (AUC) of a logistic regression prediction model. This expression is used to demonstrate how variation in measurements across settings affects the out-of-sample discriminative ability of a prediction model. We illustrate these findings with a diagnostic prediction model using example data of patients suspected of having deep venous thrombosis. RESULTS: When a predictor, such as D-dimer, is measured with more noise in one setting compared to another, which we conceptualize as a difference in "classical" measurement error, the expected value of the AUC decreases. In contrast, constant, "structural" measurement error does not impact on the AUC of a logistic regression model, provided the magnitude of the error is the same among cases and noncases. As the differences in measurement methods between settings (and in turn differences in measurement error structures) become more complex, it becomes increasingly difficult to predict how the AUC will differ between settings. CONCLUSION: When a prediction model is applied to a different setting to the one in which it was developed, its discriminative ability can decrease or even increase if the magnitude or structure of the errors in predictor measurements differ between the two settings. This provides an important starting point for researchers to better understand how differences in measurement methods can affect the performance of a prediction model when externally validating or implementing it in practice.

[Power, you can never have enough of it]. / Power, daar kun je nooit genoeg van krijgen.

- In randomised trials on the effects of a medical treatment, the power of the trial corresponds to the conditional probability that the conclusion of the trial will be that the treatment is effective, given a certain treatment effect.- The time to consider the power of a trial is before conducting the study, to ensure that the design of the trial is such that there is a reasonable chance of demonstrating a clinically relevant effect.

[Significance of p-values: misinterpreted and overrated]. / Significantie van p-waardes: onbegrepen en overschat.

- An often provided interpretation of a significant p-value (p < 0.05) is that 'the probability the conclusion is incorrect, is only 5%'. This interpretation is incorrect.- It can be shown that for observational studies, in case of p < 0.05, the probability of a false positive signal is around 50%. This means that significant p-values give us much less certainty about the reliability of a conclusion than we like to believe.- Much would be gained already if the emphasis on p-values would be replaced by: (a) an estimation of the effect size in combination with the corresponding statistical uncertainty (represented by the confidence interval), (b) an assessment of the clinical relevance of that effect, and

[Added value of observational studies in surgery: the hierarchical structure of study designs requires a more refined approach]. / Meerwaarde van observationeel onderzoek in chirurgie.

The randomised placebo-controlled trial (RCT) is the gold standard for the evaluation of medical interventions. Observational studies, on the other hand, usually do not get much credit. For studies investigating surgical interventions this does not always seem entirely justified. A more refined approach might be needed for the often-used hierarchical structure of research designs. Instead of a strict separation of results from RCTs and other designs, results of the different designs should rather be regarded as complementary to each other when evaluating surgical interventions in traumatology.

[On the timing of randomised studies: the sooner initiated, the better?] / Over de timing van gerandomiseerde trials.

The study design for randomised double-blind studies is powerful. Randomisation and blinding ensure that the groups that are compared are truly exchangeable. Any differences in health outcomes can be attributed rightfully to the one aspect on which the study groups differ: the treatment. In this commentary, we argue that the use of this powerful study design at the wrong moment can lead to undesirable situations.

The effects of misclassification in routine healthcare databases on the accuracy of prognostic prediction models: a case study of the CHA2DS2-VASc score in atrial fibrillation.

BACKGROUND: Research on prognostic prediction models frequently uses data from routine healthcare. However, potential misclassification of predictors when using such data may strongly affect the studied associations. There is no doubt that such misclassification could lead to the derivation of suboptimal prediction models. The extent to which misclassification affects the validation of existing prediction models is currently unclear.We aimed to quantify the amount of misclassification in routine care data and its effect on the validation of the existing risk prediction model. As an illustrative example, we validated the CHA2DS2-VASc prediction rule for predicting mortality in patients with atrial fibrillation (AF). METHODS: In a prospective cohort in general practice in the Netherlands, we used computerized retrieved data from the electronic medical records of patients known with AF as index predictors. Additionally, manually collected data after scrutinizing all complete medical files were used as reference predictors. Comparing the index with the reference predictors, we assessed misclassification in individual predictors by calculating Cohen's kappas and other diagnostic test accuracy measures. Predictive performance was quantified by the c-statistic and by determining calibration of multivariable models. RESULTS: In total, 2363 AF patients were included. After a median follow-up of 2.7 (IQR 2.3-3.0) years, 368 patients died (incidence rate 6.2 deaths per 100 person-years). Misclassification in individual predictors ranged from substantial (Cohen's kappa 0.56 for prior history of heart failure) to minor (kappa 0.90 for a history of type 2 diabetes). The overall model performance was not affected when using either index or reference predictors, with a c-statistic of 0.684 and 0.681, respectively, and similar calibration. CONCLUSION: In a case study validating the CHA2DS2-VASc prediction model, we found substantial predictor misclassification in routine healthcare data with only limited effect on overall model performance. Our study should be repeated for other often applied prediction models to further evaluate the usefulness of routinely available healthcare data for validating prognostic models in the presence of predictor misclassification.

Quality of life from a randomized trial of open and endovascular repair for abdominal aortic aneurysm.

BACKGROUND: Long-term survival is similar after open or endovascular repair of abdominal aortic aneurysm. Few data exist on the effect of either procedure on long-term health-related quality of life (HRQoL) and health status. METHODS: Patients enrolled in a multicentre randomized clinical trial (DREAM trial; 2000-2003) in Europe of open repair versus endovascular repair (EVAR) of abdominal aortic aneurysm were asked to complete questionnaires on health status and HRQoL. HRQoL scores were assessed at baseline and at 13 time points thereafter, using generic tools, the Medical Outcomes Study 36-Item Short-Form Health Survey (SF-36®) and EuroQol 5D (EQ-5D™). Physical (PCS) and mental component summary scores were also calculated. Follow-up was 5 years. RESULTS: Some 332 of 351 patients enrolled in the trial returned questionnaires. More than 70 per cent of questionnaires were returned at each time point. Both surgical interventions had a short-term negative effect on HRQoL and health status. This was less severe in the EVAR group than in the open repair group. In the longer term the physical domains of SF-36® favoured open repair: mean difference in PCS score between open repair and EVAR -1·98 (95 per cent c.i. -3·56 to -0·41). EQ-5D™ descriptive and EQ-5D™ visual analogue scale scores for open repair were also superior to those for EVAR after the initial 6-week interval: mean difference -0·06 (-0·10 to -0·02) and -4·09 (-6·91 to -1·27) respectively. CONCLUSION: In this study EVAR appeared to be associated with less severe disruption to HRQoL and health status in the short term. However, during longer-term follow-up to 5 years, patients receiving open repair appeared to have improved quality of life and health status.