Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity.

BACKGROUND: It is often stated that external validity is not sufficiently considered in the assessment of clinical studies. Although tools for its evaluation have been established, there is a lack of awareness of their significance and application. In this article, a comprehensive checklist is presented addressing these relevant criteria. METHODS: The checklist was developed by listing the most commonly used assessment criteria for clinical studies. Additionally, specific lists for individual applications were included. The categories of biases of internal validity (selection, performance, attrition and detection bias) correspond to structural, treatment-related and observational differences between the test and control groups. Analogously, we have extended these categories to address external validity and model validity, regarding similarity between the study population/conditions and the general population/conditions related to structure, treatment and observation. RESULTS: A checklist is presented, in which the evaluation criteria concerning external validity and model validity are systemized and transformed into a questionnaire format. CONCLUSION: The checklist presented in this article can be applied to both planning and evaluating of clinical studies. We encourage the prospective user to modify the checklists according to the respective application and research question. The higher expenditure needed for the evaluation of clinical studies in systematic reviews is justified, particularly in the light of the influential nature of their conclusions on therapeutic decisions and the creation of clinical guidelines.

Use of complementary medicine in Switzerland.

OBJECTIVE: Within the framework of the Swiss governmental Program of Evaluation of Complementary Medicine (PEK) we assessed the prevalence, use, perceived effectiveness and appreciation of complementary medicine (CAM) in Switzerland, according to published surveys. MATERIALS AND METHODS: Search was performed through electronic databases, by hand-searching and by contacting experts at universities, hospitals, health insurances, patient organizations and pharmaceutical companies. RESULTS: Surveys were carried out among the general population (40%), physicians (20%), hospitalized patients (30%) and obstetric institutions (5%). The number of publications increased strongly between 1981 and 2004. The mean +/- SD prevalence (use) of CAM is 49 +/- 22% and varies depending on the survey's topic and the population group interviewed. The acceptance, appreciation or demand for CAM among individuals specifically interviewed on CAM is 91 +/- 6%. When asked about favored general improvements in healthcare, 6.5% of the individuals spontaneously mentioned CAM. CAM therapies are considered to be effective by the majority of CAM users and by about 40% of cancer patients using CAM. Approximately 50% of the population stated a preference for hospitals that also provide CAM. 85% of the population wishes the costs for CAM to be covered by the basic health insurance. CONCLUSION: Approximately half of the Swiss population has used CAM. CAM treatment is considered to be effective by the majority of CAM users. About 50% of the population would prefer hospitals that also provide CAM therapies and the majority of the population wishes the cost for CAM therapies to be covered by basic health insurance.

Effectiveness, safety and cost-effectiveness of homeopathy in general practice - summarized health technology assessment.

INTRODUCTION: The Health Technology Assessment report on effectiveness, cost-effectiveness and appropriateness of homeopathy was compiled on behalf of the Swiss Federal Office for Public Health (BAG) within the framework of the 'Program of Evaluation of Complementary Medicine (PEK)'. MATERIALS AND METHODS: Databases accessible by Internet were systematically searched, complemented by manual search and contacts with experts, and evaluated according to internal and external validity criteria. RESULTS: Many high-quality investigations of pre-clinical basic research proved homeopathic high-potencies inducing regulative and specific changes in cells or living organisms. 20 of 22 systematic reviews detected at least a trend in favor of homeopathy. In our estimation 5 studies yielded results indicating clear evidence for homeopathic therapy. The evaluation of 29 studies in the domain 'Upper Respiratory Tract Infections/Allergic Reactions' showed a positive overall result in favor of homeopathy. 6 out of 7 controlled studies were at least equivalent to conventional medical interventions. 8 out of 16 placebo-controlled studies were significant in favor of homeopathy. Swiss regulations grant a high degree of safety due to product and training requirements for homeopathic physicians. Applied properly, classical homeopathy has few side-effects and the use of high-potencies is free of toxic effects. A general health-economic statement about homeopathy cannot be made from the available data. CONCLUSION: Taking internal and external validity criteria into account, effectiveness of homeopathy can be supported by clinical evidence and professional and adequate application be regarded as safe. Reliable statements of cost-effectiveness are not available at the moment. External and model validity will have to be taken more strongly into consideration in future studies.

Traditional Chinese medicine (phytotherapy): Health Technology Assessment report - selected aspects.

OBJECTIVE: A summary of main aspects from a Health Technology Assessment report on Traditional Chinese Medicine (TCM) in Switzerland concerning effectiveness and safety is given. MATERIALS AND METHODS: Literature search was performed through 13 databases, by scanning reference lists of articles and by contacting experts. Assessed were quality of documentation, internal and external validity. RESULTS: Effectiveness: 43 articles concerning 'gastrointestinal tract and liver' were assessed. The studies covering 7,436 patients were undertaken in China (35), Japan (3), USA (2) and Australia (3); 33/43 being controlled studies. 34/40 show significantly better results in the TCM-treated group. A comparison of studies on results of treatment based on a diagnosis according to TCM criteria and studies on results of treatment according to Western diagnosis shows that treatment based on TCM diagnosis improves the result. The comparison of treatment by individual medication and standard medication showed a trend in favor of individual medication. SAFETY: TCM training and practice for physicians in Switzerland are officially regulated. Side effects occur, but no severe effects have been registered up to now in Switzerland. TCM medicinals are imported; admission regulations are being installed. Problems due to production abroad, Internet trade, self-medication or admixtures are possible. CONCLUSION: The evaluation of the literature search provides evidence for a basic clinical effectiveness of TCM therapy. Severe side effects were not observed in Switzerland. Regulations for trading and use of medicinals prevent treatment risks. Further clinical studies in a Western context are required.

Complementary and alternative medicine costs - a systematic literature review.

OBJECTIVE: The aim of this literature review, performed within the framework of the Swiss governmental Program of Evaluation of Complementary Medicine (PEK), was to investigate costs of complementary and alternative medicine (CAM). MATERIALS AND METHODS: A systematic literature search was conducted in 11 electronic databases. All retrieved titles and reference lists were also hand-searched. RESULTS: 38 publications were found: 23 on CAM of various definitions (medical and non-medical practitioners, over-the-counter products), 13 on homeopathy, 2 on phytotherapy. Studies investigated different kinds of costs (direct or indirect) and used different methods (prospective or retrospective questionnaires, data analyses, cost-effectiveness models). Most studies report 'out of pocket' costs, because CAM is usually not covered by health insurance. Costs per CAM-treatment / patient / month were AUD 7-66, CAD 250 and GBP 13.62 +/- 1.61. Costs per treatment were EUR 205 (range: 15-1,278), USD 414 +/- 269 and USD 1,127. In two analyses phytotherapy proved to be cost-effective. One study revealed a reduction of 1.5 days of absenteeism from work in the CAM group compared to conventionally treated patients. Another study, performed by a health insurance company reported a slight increase in direct costs for CAM. Costs for CAM covered by insurance companies amounted to approximately 0.2-0.5% of the total healthcare budget (Switzerland, 2003). Publications had several limitations, e.g. efficacy of therapies was rarely reported. As compared to conventional patients, CAM patients tend to cause lower costs. CONCLUSION: Results suggest lower costs for CAM than for conventional patients, but the limited methodological quality lowers the significance of the available data. Further well-designed studies and models are required.

Development and validation of the Economic Assessment of Glycemic Control and Long-Term Effects of diabetes (EAGLE) model.

BACKGROUND: The Economic Assessment of Glycemic control and Long-term Effects of diabetes (EAGLE) model was developed to provide a flexible and comprehensive tool for the simulation of the long-term effects of diabetes treatment and related costs in type 1 and type 2 diabetes. METHODS: EAGLE simulations are based on risk equations, which were developed using published data from several large studies including the Diabetes Control and Complications Trial, the United Kingdom Prospective Diabetes Study, and the Wisconsin Epidemiological Study of Diabetic Retinopathy. Risk equations for the probability of complications (including hypoglycemia, retinopathy, macular edema, end-stage renal disease, neuropathy, diabetic foot syndrome, myocardial infarction, and stroke) were based on regression analyses, using linear, exponential, and quadratic regression formulae. Subsequent cost calculations are made from the simulated event rates. Internal validation of the EAGLE model was completed by comparing simulated event rates with the published event rates used as the basis for the model. RESULTS: EAGLE provides microsimulations of virtual patient cohorts for type 1 and type 2 diabetes over n years in 1-year cycles. Complications include microvascular and macrovascular events and death, which are calculated over time as cumulative incidences. Glycosylated hemoglobin levels over time are simulated in relation to treatment regimen. Internal validation demonstrated that each mean event rate simulated by EAGLE overlapped with the published mean event (within a range of +/-10%). CONCLUSIONS: The EAGLE model is an evidence-based, internally valid tool for the assessment of the long-term effects of diabetes treatment and related costs.

Costs of irritable bowel syndrome in the UK and US.

Irritable bowel syndrome (IBS) is one of the most common functional gastrointestinal disorders, with an estimated prevalence rate in the general population of 10-15% in industrialised countries. Although IBS is not a life-threatening disease, it contributes significantly to a large segment of healthcare resource consumption. This review provides an overview of studies addressing the direct and indirect costs of IBS in the US and the UK. A systematic literature search was conducted in MEDLINE and the Cochrane library; additionally, all reference lists covering the years from 1960 to May 2004 were scanned. Twenty-four publications for the US and the UK, published between 1991 and 2003, were identified: 6 were excluded, 18 were included. Data for the UK, US and UK + US were reported in 5, 11 and 2 publications, respectively. Total direct cost estimates per patient per year ranged from US 348 dollars to US 8750 dollars (calculated for year 2002). The average number of days off work per year because of IBS was between 8.5 and 21.6; indirect costs ranged from US 355 dollars to US 3344 dollars. The total costs and cost components of IBS are influenced by several factors: features of the investigated patient group (age, limitation to healthcare seekers or all IBS patients, comorbidity, severity of symptoms), database used, method of data collection (retrospective or prospective, varying cost components, time-point of data collection in relation to index-date of IBS diagnosis, method of cost calculation [incidence or prevalence based]) and different healthcare systems in the US and the UK. These factors led to the incomparability of published data, thus no comprehensive picture can be drawn of the total costs related to IBS in the UK and US. Data underline the magnitude of the economic impact of IBS in the UK and US, which is increased by a factor of 1.1-6.0, compared with matched non-IBS control groups. IBS contributes both direct and indirect costs to the total healthcare bill. Further studies should take influencial factors into account and report related data carefully in order to provide useful and comparable published cost data. Additionally, further research on the cost effectiveness of diagnostic procedures and therapies in IBS is required to define strategies to help IBS patients improve their quality of life and reduce related costs.

International variation in resource utilisation and treatment costs for rheumatoid arthritis: a systematic literature review.

Recent years have witnessed substantial progress in understanding the cost implications of rheumatoid arthritis (RA). To assess the divergent methodologies and their impact on the resulting cost analyses in RA, we conducted a systematic literature review to summarise the scientific evidence of RA-induced costs. Sixty-five reviews, models or cost analyses on the burden of illness and general costs associated with RA were identified. They covered the US, Canada, Sweden, the UK, The Netherlands, Germany and Finland. Twenty-four cost analyses provided appropriate data about direct and/or indirect costs. Each study was summarised separately. Costs were discounted to 2003 and converted to US dollars. The costs per RA-year ranged from USD 1503 to USD 16,514. However, each study has to be interpreted individually, with consideration given to the study population, indication, age of the study, database used, type of therapy, setting, level of cost differentiation and data derivation. Health technology assessment reports offer sufficient space to adequately describe the composite parts and restrictive elements of different methodological approaches and analyses.