Ethical challenges for Health Technology Assessment (HTA) in the evolving evidence landscape.

Since its inception, Health Technology Assessment (HTA) has typically determined the value of a technology by collecting information derived from randomized clinical trials (RCTs), in line with the principles of evidence-based medicine (EBM). However, data from RCTs did not constitute the sole source of information, as other types of evidence (such as primary qualitative research) have often been utilized. Recent advances in both generating and collecting other types of evidence are broadening the landscape of evidence, adding complexity to the discussion of "robustness of evidence." What are the consequences of these recent developments for the methodology and conduct of HTA, the HTA community, and its ethical commitments? The aim of this article is to explore some ethical challenges that are emerging in the current evolving evidence landscape, particularly changes in evidence generation and collection (e.g., diversification of data sources), and shifting standards of evidence in the field of HTA (e.g., increasing acceptability of evidence that is thought of as lower quality). Our conclusion is that deciding how to best maintain trustworthiness is common to all these issues.

Challenges and Opportunities for Deliberative Processes for Healthcare Decision-Making Comment on "Evidence-Informed Deliberative Processes for Health Benefit Package Design - Part II: A Practical Guide".

The second edition of the practical guide for evidence-informed deliberative processes (EDPs) is an important addition to the growing guidance on deliberative processes supporting priority setting in healthcare. While the practical guide draws on an extensive amount of information collected on established and developing processes within a range of countries, EDPs present health technology assessment (HTA) bodies with several challenges. (1) Basing recommendations on current processes that have not been well-evaluated and that have changed over time may lead to weaker legitimacy than desired. (2) The requirement for social learning among stakeholders may require increased resourcing and blur the boundary between moral deliberation and political negotiation. (3) Robust evaluation should be based on an explicit theory of change, and some process outcomes may be poor guides to overall improvement of EDPs. This comment clarifies and reinforces the recommendations provided in the practical guide.

Core competencies for ethics experts in health technology assessment.

OBJECTIVES: There is no consensus on who might be qualified to conduct ethical analysis in the field of health technology assessment (HTA). Is there a specific expertise or skill set for doing this work? The aim of this article is to (i) clarify the concept of ethics expertise and, based on this, (ii) describe and specify the characteristics of ethics expertise in HTA. METHODS: Based on the current literature and experiences in conducting ethical analysis in HTA, a group of members of the Health Technology Assessment International (HTAi) Interest Group on Ethical Issues in HTA critically analyzed the collected information during two face-to-face workshops. On the basis of the analysis, working definitions of "ethics expertise" and "core competencies" of ethics experts in HTA were developed. This paper reports the output of the workshop and subsequent revisions and discussions online among the authors. RESULTS: Expertise in a domain consists of both explicit and tacit knowledge and is acquired by formal training and social learning. There is a ubiquitous ethical expertise shared by most people in society; nevertheless, some people acquire specialist ethical expertise. To become an ethics expert in the field of HTA, one needs to acquire general knowledge about ethical issues as well as specific knowledge of the ethical domain in HTA. The core competencies of ethics experts in HTA consist of three fundamental elements: knowledge, skills, and attitudes. CONCLUSIONS: The competencies described here can be used by HTA agencies and others involved in HTA to call attention to and strengthen ethical analysis in HTA.

Principles for deliberative processes in health technology assessment.

Deliberative processes are a well-established part of health technology assessment (HTA) programs in a number of high- and middle-income countries, and serve to combine complex sets of evidence, perspectives, and values to support open, transparent, and accountable decision making. Nevertheless, there is little documentation and research to inform the development of effective and efficient deliberative processes, and to evaluate their quality. This article summarizes the 2020 HTAi Global Policy Forum (GPF) discussion on deliberative processes in HTA.Through a combination of small and large group discussion and successive rounds of polling, the GPF members reached strong agreement on three core principles for deliberative processes in HTA: transparency, inclusivity, and impartiality. In addition, discussions revealed other important principles, such as respect, reviewability, consistency, and reasonableness, that may supplement the core set. A number of associated supporting actions for each of the principles are also described in order to make each principle realizable in a given HTA setting. The relative importance of the principles and actions are context-sensitive and must be considered in light of the political, legislative, and operational factors that may influence the functioning of any particular HTA environment within which the deliberative process is situated. The paper ends with suggested concrete next steps that HTA agencies, researchers, and stakeholders might take to move the field forward. The proposed principles and actions, and the next steps, provide a springboard for further research and better documentation of important aspects of deliberation that have historically been infrequently studied.

Ethical Challenges Related to Patient Involvement in Health Technology Assessment.

Including information and values from patients in HTA has the potential to improve both the process and outcomes of health technology policy decisions. Accordingly, funding and structural incentives to include patients in HTA activities have increased over the past several years. Unfortunately, these incentives have not yet been accompanied by a corresponding increase in resources, time, or commitment to responsiveness. In this Perspectives piece, we reflect on our collective experiences participating in, conducting, and overseeing patient engagement activities within HTA to highlight the ethical challenges associated with this area of activity. While we remain committed to the idea that patient engagement activities strengthen the findings, relevance, and legitimacy of health technology policy, we are deeply concerned about the potential for these activities to do ethical harm. We use this analysis to call for action to introduce strong protections against ethical violations that may harm patients participating in HTA engagement activities.

Identifying the Need for Good Practices in Health Technology Assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA.

The systematic use of evidence to inform healthcare decisions, particularly health technology assessment (HTA), has gained increased recognition. HTA has become a standard policy tool for informing decision makers who must manage the entry and use of pharmaceuticals, medical devices, and other technologies (including complex interventions) within health systems, for example, through reimbursement and pricing. Despite increasing attention to HTA activities, there has been no attempt to comprehensively synthesize good practices or emerging good practices to support population-based decision-making in recent years. After the identification of some good practices through the release of the ISPOR Guidelines Index in 2013, the ISPOR HTA Council identified a need to more thoroughly review existing guidance. The purpose of this effort was to create a basis for capacity building, education, and improved consistency in approaches to HTA-informed decision-making. Our findings suggest that although many good practices have been developed in areas of assessment and some other key aspects of defining HTA processes, there are also many areas where good practices are lacking. This includes good practices in defining the organizational aspects of HTA, the use of deliberative processes, and measuring the impact of HTA. The extent to which these good practices are used and applied by HTA bodies is beyond the scope of this report, but may be of interest to future researchers.

Correction: Methods for Developing Evidence Reviews in Short Periods of Time: A Scoping Review.

[This corrects the article DOI: 10.1371/journal.pone.0165903.].

QUALITY ASSESSMENT OF ETHICS ANALYSES FOR HEALTH TECHNOLOGY ASSSESSMENT.

OBJECTIVES: Although consideration of ethical issues is recognized as a crucial part of health technology assessment, ethics analysis for HTA is generally perceived as methodologically underdeveloped in comparison to other HTA domains. The aim of our study is (i) to verify existing tools for quality assessment of ethics analyses for HTA, (ii) to consider some arguments for and against the need for quality assessment tools for ethics analyses for HTA, and (iii) to propose a preliminary set of criteria that could be used for assessing the quality of ethics analyses for HTA. METHODS: We systematically reviewed the literature, reviewed HTA organizations' Web sites, and solicited views from thirty-two experts in the field of ethics for HTA. RESULTS: The database and HTA agency Web site searches yielded 420 references (413 from databases, seven from HTA Web sites). No formal instruments for assessing the quality of ethics analyses for HTA purposes were identified. Thirty-two experts in the field of ethics for HTA from ten countries, who were brought together at two workshops held in Edmonton (Canada) and Cologne (Germany) confirmed the findings from the literature. CONCLUSIONS: Generating a quality assessment tool for ethics analyses in HTA would confer considerable benefits, including methodological alignment with other areas of HTA, increase in transparency and transferability of ethics analyses, and provision of common language between the various participants in the HTA process. We propose key characteristics of quality assessment tools for this purpose, which can be applied to ethics analyses for HTA purposes.

Methods for Developing Evidence Reviews in Short Periods of Time: A Scoping Review.

INTRODUCTION: Rapid reviews (RR), using abbreviated systematic review (SR) methods, are becoming more popular among decision-makers. This World Health Organization commissioned study sought to summarize RR methods, identify differences, and highlight potential biases between RR and SR. METHODS: Review of RR methods (Key Question 1 [KQ1]), meta-epidemiologic studies comparing reliability/ validity of RR and SR methods (KQ2), and their potential associated biases (KQ3). We searched Medline, EMBASE, Cochrane Library, grey literature, and checked reference lists, used personal contacts, and crowdsourcing (e.g. email listservs). Selection and data extraction was conducted by one reviewer (KQ1) or two reviewers independently (KQ2-3). RESULTS: Across all KQs, we identified 42,743 citations through the literature searches. KQ1: RR methods from 29 organizations were reviewed. There was no consensus on which aspects of the SR process to abbreviate. KQ2: Studies comparing the conclusions of RR and SR (n = 9) found them to be generally similar. Where major differences were identified, it was attributed to the inclusion of evidence from different sources (e.g. searching different databases or including different study designs). KQ3: Potential biases introduced into the review process were well-identified although not necessarily supported by empirical evidence, and focused mainly on selective outcome reporting and publication biases. CONCLUSION: RR approaches are context and organization specific. Existing comparative evidence has found similar conclusions derived from RR and SR, but there is a lack of evidence comparing the potential of bias in both evidence synthesis approaches. Further research and decision aids are needed to help decision makers and reviewers balance the benefits of providing timely evidence with the potential for biased findings.

Ethics expertise for health technology assessment: a Canadian national survey.

OBJECTIVES: The aim of this study was to identify individuals with expertise in ethics analysis in Canada, who might contribute to health technology assessment (HTA); to gauge these individuals' familiarity with, and experience participating in, the production of HTA. METHODS: A contact list was developed using the Canadian Bioethics Society membership list and faculty listings of Canadian universities, bioethics centers, and health agencies. An eighteen-question email survey was distributed to potential respondents to collect data on demographic information, education and work experience in applied ethics, and involvement in HTA. RESULTS: The survey response rate was 52.8 percent (350/663). Respondents worked primarily in academic institutions (50.4 percent) or hospitals (15.4 percent). Many respondents (83.1 percent) had education, formal training, or work-related experience in practical ethics related to health care, with many having a doctorate (34.5 percent) or master's degree (19.0 percent). One quarter (24.5 percent; n = 87) of respondents indicated they had been involved in an analysis of ethical issues for HTA. Almost two-thirds (65.4 percent; n = 165) of those who had not previously participated in ethics analysis believed they might usefully contribute to an analysis of ethical issues in HTA. Experts who have conducted ethics analysis in HTA had more than twice the odds of having education and training in ethics and a PhD than those who might contribute to ethics analysis. CONCLUSION: Many people have contributed to ethics analysis in HTA in Canada, and more are willing to do so. Given the absence of a reliable credential for ethics expertise, HTA producers should exercise caution when enlisting ethics experts.

Revealing and acknowledging value judgments in health technology assessment.

BACKGROUND: Although value issues are increasingly addressed in health technology assessment (HTA) reports, HTA is still seen as a scientific endeavor and sometimes contrasted with value judgments, which are considered arbitrary and unscientific. This article aims at illustrating how numerous value judgments are at play in the HTA process, and why it is important to acknowledge and address value judgments. METHODS: A panel of experts involved in HTA, including ethicists, scrutinized the HTA process with regard to implicit value judgments. It was analyzed whether these value judgments undermine the accountability of HTA results. The final results were obtained after several rounds of deliberation. RESULTS: Value judgments are identified before the assessment when identifying and selecting health technologies to assess, and as part of assessment. They are at play in the processes of deciding on how to select, frame, present, summarize or synthesize information in systematic reviews. Also, in economic analysis, value judgments are ubiquitous. Addressing the ethical, legal, and social issues of a given health technology involves moral, legal, and social value judgments by definition. So do the appraisal and the decision-making process. CONCLUSIONS: HTA by and large is a process of value judgments. However, the preponderance of value judgments does not render HTA biased or flawed. On the contrary they are basic elements of the HTA process. Acknowledging and explicitly addressing value judgments may improve the accountability of HTA.

Testing a tool for the classification of study designs in systematic reviews of interventions and exposures showed moderate reliability and low accuracy.

OBJECTIVES: To develop and test a study design classification tool. STUDY DESIGN: We contacted relevant organizations and individuals to identify tools used to classify study designs and ranked these using predefined criteria. The highest ranked tool was a design algorithm developed, but no longer advocated, by the Cochrane Non-Randomized Studies Methods Group; this was modified to include additional study designs and decision points. We developed a reference classification for 30 studies; 6 testers applied the tool to these studies. Interrater reliability (Fleiss' κ) and accuracy against the reference classification were assessed. The tool was further revised and retested. RESULTS: Initial reliability was fair among the testers (κ=0.26) and the reference standard raters κ=0.33). Testing after revisions showed improved reliability (κ=0.45, moderate agreement) with improved, but still low, accuracy. The most common disagreements were whether the study design was experimental (5 of 15 studies), and whether there was a comparison of any kind (4 of 15 studies). Agreement was higher among testers who had completed graduate level training versus those who had not. CONCLUSION: The moderate reliability and low accuracy may be because of lack of clarity and comprehensiveness of the tool, inadequate reporting of the studies, and variability in tester characteristics. The results may not be generalizable to all published studies, as the test studies were selected because they had posed challenges for previous reviewers with respect to their design classification. Application of such a tool should be accompanied by training, pilot testing, and context-specific decision rules.

Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma.

BACKGROUND: The Risk of Bias (RoB) tool is used to assess internal validity of randomized controlled trials (RCTs). Our objectives were to: 1) evaluate inter-rater agreement of the RoB tool; 2) determine the time to access supplemental study information; 3) compare the RoB tool with the Jadad scale and Schulz allocation concealment (AC); and 4) examine the relationship between RoB and effect estimates. METHODS: We conducted a systematic review of long-acting beta agonists (LABA) combined with inhaled corticosteroids (ICS) for adults with persistent asthma. Two reviewers independently assessed 107 trials using RoB, Jadad, and AC. One reviewer searched for study protocols. We assessed inter-rater agreement using weighted Kappa (κ) and the correlation between tools using Kendall's Tau (τ). Mean differences in effect sizes for RCTs with different RoB were calculated using inverse variance method and random effects model. RESULTS: Trials had good Jadad scores (median 4, IQR 3-4); however, 85% had unclear AC and 87% high RoB. The factor that most influenced RoB was the potential inappropriate influence of study sponsors (95% industry funded). Agreement on RoB domains was fair (κ = 0.40) to almost perfect (κ = 0.86), and moderate for overall RoB (κ = 0.41). Median time to complete RoB assessments was 21 minutes (IQR 14-27) and 12 minutes (IQR 9-16) to search for protocols. Protocols were identified for 5/42 studies (12%); in 3 cases the assessment of selective outcome reporting changed. There was low correlation between overall RoB vs. Jadad (τ = 0.04, p = 0.3) and AC (τ = -0.02, p = 0.7). Analyses comparing effect estimates and risk showed no important patterns. CONCLUSIONS: Inter-rater agreement on RoB assessments was better than previously reported suggesting that review-specific guidelines are important. The correlation between RoB and Jadad was low suggesting measurement of different constructs (risk of bias vs. quality of reporting). The extensive involvement of the pharmaceutical industry in this LABA/ICS research should raise concerns about potential overestimates of treatment effects.

Improving ethics analysis in health technology assessment.

OBJECTIVES: The aim of this study was to critically examine the current guidance for conducting ethics analysis in health technology assessment (HTA) and to offer recommendations for how to improve this practice. METHODS: MEDLINE, Philosopher's Index, and Google Scholar were searched for articles and reports using the keywords "ethics" and "health technology assessment" and related terms. Bibliographies of all relevant articles were also examined for additional references. A philosophical analysis of the existing guidance was conducted. RESULTS: We offer three recommendations for improving ethics analysis in HTA. First, ethical and legal issues must be clearly separated so that all policy-relevant questions that the technology raises can be considered clearly and systematically. Second, analysts must make better use of ethics theory and discuss better how particular theoretical approaches and associated analytic tools are selected to make transparent which alternative approaches were considered and why they were rejected. Third, the necessity for philosophical expertise to adequately conduct ethics analysis needs to be acknowledged. CONCLUSIONS: To act on these recommendations for ethics analysis, we offer these three steps forward: acknowledge and use relevant expertise, further develop models for conducting and reporting ethics analyses, and make use of untapped resources in the literature.

Tracking emergency department overcrowding in a tertiary care academic institution.

Despite the release of a national report describing key markers of emergency department (ED) overcrowding, limited linear data using these markers have been published. We sought to report the degree and trends of ED overcrowding in a typical academic hospital and to highlight some of the key markers of ED patient flow and care. We conducted a prospective study in a large Canadian urban tertiary care teaching hospital that receives approximately 55,000 annual adult ED visits. A database captured demographic and real-time process of care data for each patient from 2000 to 2007. Descriptive data are reported using Canadian Triage and Acuity Scale (CTAS) scores. Over the study period, the ED patient visit volume and presentation times remained predictable. Emergent cases (CTAS levels 1-2) doubled from 8 to 16.6%, and urgent cases (CTAS level 3) increased from 40.2 to 50.3%. Moreover, semi-urgent presentations (CTAS level 4) decreased from 42.4 to 28.8%, and non-urgent cases (CTAS level 5) dropped from 9.4 to 4.3%. The median wait time from triage to bed location increased from two minutes (inter-quartile range [IQR] 1, 46) in 2000 to 27 minutes (IQR 2, 110) in 2007, while the median time from bed location to physician remained constant (29 minutes in 2001 versus 28 minutes in 2007). Overall, admissions increased from 20.4 to 23%. Semi-urgent and non-urgent admissions dropped from 11.5 to 7.4% and 3.2 to 1.8%, respectively. Admitted patients "boarding" in the ED increased from 70,955 hours in 2002 to 118,741 hours in 2007, while the number of emergent and urgent patients leaving without being seen increased by more than 400%. ED overcrowding in a tertiary care hospital is primarily a result of access block due to boarding admitted patients, a situation that poses serious risks to the majority of patients who have emergent or urgent conditions that cannot be managed appropriately in the waiting room.

Clinical trials of meditation practices in health care: characteristics and quality.

OBJECTIVE: To provide a descriptive overview of the clinical trials assessing meditation practices for health care. DESIGN: Systematic review of the literature. Comprehensive searches were conducted in 17 electronic bibliographic databases through September 2005. Other sources of potentially relevant studies included hand searches, reference tracking, contacting experts, and gray literature searches. Included studies were clinical trials with 10 or more adult participants using any meditation practice, providing quantitative data on health-related outcomes, and published in English. Two independent reviewers assessed study relevance, extracted the data, and assessed the methodological quality of the studies. RESULTS: Four hundred clinical trials on meditation (72% described as randomized) were included in the review (publication years 1956-2005). Five broad categories of meditation practices were identified: mantra meditation, mindfulness meditation, yoga, t'ai chi, and qigong. The three most studied clinical conditions were hypertension, miscellaneous cardiovascular diseases, and substance abuse. Psychosocial measures were the most frequently reported outcomes. Outcome measures of psychiatric and psychological symptoms dominate the outcomes of interest. Overall, the methodological quality of clinical trials is poor, but has significantly improved over time by 0.014 points every year (95% CI, 0.005, 0.023). CONCLUSIONS: Most clinical trials on meditation practices are generally characterized by poor methodological quality with significant threats to validity in every major quality domain assessed. Despite a statistically significant improvement in the methodological quality over time, it is imperative that future trials on meditation be rigorous in design, execution, analysis, and the reporting of results.

Frequency, determinants and impact of overcrowding in emergency departments in Canada: a national survey.

Several reports have documented the prevalence and severity of emergency department (ED) overcrowding at specific hospitals or cities in Canada; however, no study has examined the issue at a national level. A 54-item, self-administered, postal and web-based questionnaire was distributed to 243 ED directors in Canada to collect data on the frequency, impact and factors associated with ED overcrowding. The survey was completed by 158 (65% response rate) ED directors, 62% of whom reported overcrowding as a major or severe problem during the past year. Directors attributed overcrowding to a variety of issues including a lack of admitting beds (85%), lack of acute care beds (74%) and the increased length of stay of admitted patients in the ED (63%). They perceived ED overcrowding to have a major impact on increasing stress among nurses (82%), ED wait times (79%) and the boarding of admitted patients in the ED while waiting for beds (67%). Overcrowding is not limited to large urban centres; nor is it limited to academic and teaching hospitals. The perspective of ED directors reinforces the need for further examination of effective policies and interventions to reduce ED overcrowding.

Key indicators of overcrowding in Canadian emergency departments: a Delphi study.

OBJECTIVE: To identify the level of consensus among a group of Canadian emergency department (ED) experts on the importance of a set of indicators to document ED overcrowding. METHODS: A 2-round Delphi survey was conducted from February 2005 to April 2005, with a multidisciplinary group of 38 Canadian experts in various aspects of ED operations who rated the relevance of 36 measures and ranked their relative importance as indicators of ED overcrowding. RESULTS: The response rates for the first and second rounds were 84% and 87%, respectively. The most important indicator identified by the experts was the percentage of the ED occupied by inpatients (mean on a 7-point Likert-type scale 6.53, standard deviation [SD] 0.80). The other 9 indicators, in order of the importance attributed, were the total number of ED patients (mean 6.35, SD 0.75), the total time in the ED (mean 6.16, SD 1.04), the percentage of time that the ED was at or above capacity (mean 6.16, SD 1.08), the overall bed occupancy (mean 6.19, SD 0.93), the time from bed request to bed assignment (mean 6.06, SD 1.08), the time from triage to care (mean 5.84, SD 1.08) the physician satisfaction (mean 5.84, SD 1.22), the time from bed availability to ward transfer (mean 5.53, SD 1.72) and the number of staffed acute care beds (mean 5.53, SD 1.57). CONCLUSION: Ten clinically important measures were prioritized by the participants as relevant indicators of ED overcrowding. Indicators derived from consensus techniques have face validity, but their metric properties must be tested to ensure their effectiveness for identifying ED overcrowding in different settings.

Meditation practices for health: state of the research.

OBJECTIVES: To review and synthesize the state of research on a variety of meditation practices, including: the specific meditation practices examined; the research designs employed and the conditions and outcomes examined; the efficacy and effectiveness of different meditation practices for the three most studied conditions; the role of effect modifiers on outcomes; and the effects of meditation on physiological and neuropsychological outcomes. DATA SOURCES: Comprehensive searches were conducted in 17 electronic databases of medical and psychological literature up to September 2005. Other sources of potentially relevant studies included hand searches, reference tracking, contact with experts, and gray literature searches. REVIEW METHODS: A Delphi method was used to develop a set of parameters to describe meditation practices. Included studies were comparative, on any meditation practice, had more than 10 adult participants, provided quantitative data on health-related outcomes, and published in English. Two independent reviewers assessed study relevance, extracted the data and assessed the methodological quality of the studies. RESULTS: Five broad categories of meditation practices were identified (Mantra meditation, Mindfulness meditation, Yoga, Tai Chi, and Qi Gong). Characterization of the universal or supplemental components of meditation practices was precluded by the theoretical and terminological heterogeneity among practices. Evidence on the state of research in meditation practices was provided in 813 predominantly poor-quality studies. The three most studied conditions were hypertension, other cardiovascular diseases, and substance abuse. Sixty-five intervention studies examined the therapeutic effect of meditation practices for these conditions. Meta-analyses based on low-quality studies and small numbers of hypertensive participants showed that TM(R), Qi Gong and Zen Buddhist meditation significantly reduced blood pressure. Yoga helped reduce stress. Yoga was no better than Mindfulness-based Stress Reduction at reducing anxiety in patients with cardiovascular diseases. No results from substance abuse studies could be combined. The role of effect modifiers in meditation practices has been neglected in the scientific literature. The physiological and neuropsychological effects of meditation practices have been evaluated in 312 poor-quality studies. Meta-analyses of results from 55 studies indicated that some meditation practices produced significant changes in healthy participants. CONCLUSIONS: Many uncertainties surround the practice of meditation. Scientific research on meditation practices does not appear to have a common theoretical perspective and is characterized by poor methodological quality. Firm conclusions on the effects of meditation practices in healthcare cannot be drawn based on the available evidence. Future research on meditation practices must be more rigorous in the design and execution of studies and in the analysis and reporting of results.

Data collection on patients in emergency departments in Canada.

OBJECTIVE: Relatively little is known about the ability of Canadian emergency departments (EDs) and the federal, provincial and territorial governments to quantify ED activity. The objectives of this study were to determine the use of electronic patient data in Canadian EDs, the accessibility of provincial data on ED visits, and to identify the data elements and current methods of ED information system (EDIS) data collection nationally. METHODS: Surveys were conducted of the following 3 groups: 1) all ED directors of Canadian hospitals located in communities of >10,000 people, 2) all electronic EDIS vendors, and 3) representatives from the ministries of health from 13 provincial and territorial jurisdictions who had knowledge of ED data collection. RESULTS: Of the 243 ED directors contacted, 158 completed the survey (65% response rate) and 39% of those reported using an electronic EDIS. All 11 EDIS vendor representatives responded. Most of the vendors provide a similar package of basic EDIS options, with add-on features. All 13 provincial or territorial government representatives completed the survey. Nine (69%) provinces and territories collect ED data, however the source of this information varies. Five provinces and territories collect triage data, and 3 have a comprehensive, jurisdiction-wide, population-based ED database. Thirty-nine percent of EDs in larger Canadian communities track patients using electronic methods. A variety of EDIS vendor options are available and used in Canada. CONCLUSION: The wide variation in methods and in data collected presents serious barriers to meaningful comparison of ED services across the country. It is little wonder that the majority of information regarding ED overcrowding in Canada is anecdotal, when the collection of this critical health information is so variable. There is an urgent need to place the collection of ED information on the provincial and national agenda and to ensure that the collection of this information consistent, comprehensive and mandatory.