A novel approach to sharing all available information from funded health research: the NIHR Journals Library.

BACKGROUND: Relevant information on health research must be made publicly available in an accurate, timely and accessible manner if evidence is to inform practice and benefit patient care. Failure to publish research information represents a significant waste of research funds. However, recent studies have demonstrated that non-publication and selective or biased reporting remains a significant problem. The role of online publications in rectifying these issues by providing open access to study information is increasingly recognised. OBJECTIVE: This paper details a novel approach to publishing research information developed by the National Institute for Health Research (NIHR), a major funder of health research in the United Kingdom. The NIHR has enhanced its Journals Library ( www.journalslibrary.nihr.ac.uk ), providing an online repository of information from research funded through five programmes. We describe how the NIHR Journals Library provides a 'thread' of relevant information for each study, including protocols, participant information sheets, data linkages, final reports, publications and diverse knowledge products. We also discuss the Library as a 'living' resource, one that is updated as each study progresses from inception to completion. Finally, we consider the implications of the Library for the NIHR, other journals and research teams submitting information. CONCLUSION: Openly publishing information from funded research in the NIHR Journals Library serves as a model of knowledge sharing, maximising return on investment and enhancing the usability and replicability of research findings for different evidence-user communities. The Library also supports wider 'research on research' ambitions, enabling users to interrogate the repository of NIHR-funded studies, enhancing the understanding of research commissioning, design, dissemination and impact. Video abstract: www.youtube.com/watch?v=8H03uxN_iTE .

Clinical trial metadata: defining and extracting metadata on the design, conduct, results and costs of 125 randomised clinical trials funded by the National Institute for Health Research Health Technology Assessment programme.

BACKGROUND: By 2011, the Health Technology Assessment (HTA) programme had published the results of over 100 trials with another 220 in progress. The aim of the project was to develop and pilot 'metadata' on clinical trials funded by the HTA programme. OBJECTIVES: The aim of the project was to develop and pilot questions describing clinical trials funded by the HTA programme in terms of it meeting the needs of the NHS with scientifically robust studies. The objectives were to develop relevant classification systems and definitions for use in answering relevant questions and to assess their utility. DATA SOURCES: Published monographs and internal HTA documents. REVIEW METHODS: A database was developed, 'populated' using retrospective data and used to answer questions under six prespecified themes. Questions were screened for feasibility in terms of data availability and/or ease of extraction. Answers were assessed by the authors in terms of completeness, success of the classification system used and resources required. Each question was scored to be retained, amended or dropped. RESULTS: One hundred and twenty-five randomised trials were included in the database from 109 monographs. Neither the International Standard Randomised Controlled Trial Number nor the term 'randomised trial' in the title proved a reliable way of identifying randomised trials. Only limited data were available on how the trials aimed to meet the needs of the NHS. Most trials were shown to follow their protocols but updates were often necessary as hardly any trials recruited as planned. Details were often lacking on planned statistical analyses, but we did not have access to the relevant statistical plans. Almost all the trials reported on cost-effectiveness, often in terms of both the primary outcome and quality-adjusted life-years. The cost of trials was shown to depend on the number of centres and the duration of the trial. Of the 78 questions explored, 61 were well answered, 33 fully with 28 requiring amendment were the analysis updated. The other 17 could not be answered with readily available data. LIMITATIONS: The study was limited by being confined to 125 randomised trials by one funder. CONCLUSIONS: Metadata on randomised controlled trials can be expanded to include aspects of design, performance, results and costs. The HTA programme should continue and extend the work reported here. FUNDING: The National Institute for Health Research HTA programme.

The completeness of intervention descriptions in published National Institute of Health Research HTA-funded trials: a cross-sectional study.

OBJECTIVES: The objective of this study was to assess whether National Institute of Health Research (NIHR) Health Technology Assessment (HTA)-funded randomised controlled trials (RCTs) published in the HTA journal were described in sufficient detail to replicate in practice. SETTING: RCTs published in the HTA journal. PARTICIPANTS: 98 RCTs published in the HTA journal up to March 2011. Completeness of the intervention description was assessed independently by two researchers using a checklist, which included assessments of participants, intensity, schedule, materials and settings. Disagreements in scoring were discussed in the team; differences were then explored and resolved. PRIMARY AND SECONDARY OUTCOME MEASURES: Proportion of trials rated as having a complete description of the intervention (primary outcome measure). The proportion of drug trials versus psychological and non-drug trials rated as having a complete description of the intervention (secondary outcome measures). RESULTS: Components of the intervention description were missing in 68/98 (69.4%) reports. Baseline characteristics and descriptions of settings had the highest levels of completeness with over 90% of reports complete. Reports were less complete on patient information with 58.2% of the journals having an adequate description. When looking at individual intervention types, drug intervention descriptions were more complete than non-drug interventions with 33.3% and 30.6% levels of completeness, respectively, although this was not significant statistically. Only 27.3% of RCTs with psychological interventions were deemed to be complete, although again these differences were not significant statistically. CONCLUSIONS: Ensuring the replicability of study interventions is an essential part of adding value in research. All those publishing clinical trial data need to ensure transparency and completeness in the reporting of interventions to ensure that study interventions can be replicated.

Time to publication for NIHR HTA programme-funded research: a cohort study.

OBJECTIVE: To assess the time to publication of primary research and evidence syntheses funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme published as a monograph in Health Technology Assessment and as a journal article in the wider biomedical literature. STUDY DESIGN: Retrospective cohort study. SETTING: Primary research and evidence synthesis projects funded by the HTA Programme were included in the cohort if they were registered in the NIHR research programmes database and was planned to submit the draft final report for publication in Health Technology Assessment on or before 9 December 2011. MAIN OUTCOME MEASURES: The median time to publication and publication at 30 months in Health Technology Assessment and in an external journal were determined by searching the NIHR research programmes database and HTA Programme website. RESULTS: Of 458 included projects, 184 (40.2%) were primary research projects and 274 (59.8%) were evidence syntheses. A total of 155 primary research projects had a completion date; the median time to publication was 23 months (26.5 and 35.5 months to publish a monograph and to publish in an external journal, respectively) and 69% were published within 30 months. The median time to publication of HTA-funded trials (n=126) was 24 months and 67.5% were published within 30 months. Among the evidence syntheses with a protocol online date (n=223), the median time to publication was 25.5 months (28 months to publication as a monograph), but only 44.4% of evidence synthesis projects were published in an external journal. 65% of evidence synthesis studies had been published within 30.0 months. CONCLUSIONS: Research funded by the HTA Programme publishes promptly. The importance of Health Technology Assessment was highlighted as the median time to publication was 9 months shorter for a monograph than an external journal article.

Assessing the international use of health technology assessments: exploring the merits of different methods when applied to the National Institute of Health Research Health Technology Assessment (NIHR HTA) programme.

OBJECTIVES: This study presents findings from a study that explores the merits of different methods for assessing the international use of UK funded research by the National Institute of Health Research Health Technology Assessment (NIHR HTA) Programme. METHODS: The study adopted an exploratory approach and used three core methods: (i) Academic use was explored through bibliometric and citation analysis of the top ten most cited health technology assessment (HTA) reports. (ii) Internet use was assessed using Webtrends software to identify the proportion of international visits of the top ten most downloaded HTA reports from January 1, 2004 to June 30, 2010. (iii) International HTA use was assessed by searching the Center for Reviews and Dissemination (CRD) HTA database to explore the citation of NIHR HTA reports in reports by non-UK HTA agencies. RESULTS: Bibliometric analysis identified published output and international citations with 41 percent of the 549 journals citing NIHR HTA reports being based in the United States. Nine of ten most downloaded reports from the NIHR HTA Web site (www.hta.ac.uk) had in excess of 50 percent of visits outside the United Kingdom. Four of five selected NIHR HTA reports were cited in twenty-eight other HTA reports, eighteen of these outside the United Kingdom. CONCLUSIONS: Assessing international use is important when exploring the uptake of research evidence. Methods used in identifying research impact, such as bibliometrics and Webtrends, are helpful in generating evidence of international use. HTA agencies should consider these techniques and international use when assessing the uptake of findings from research they undertake and/or commission.

Registration of noncommercial randomised clinical trials: the feasibility of using trial registries to monitor the number of trials.

BACKGROUND: A 2003 survey suggested the number of noncommercial trials in the UK was declining. Formation of the NIHR in 2006 and increased research spending by the Department of Health may have increased the number of noncommercial trials but no data are available. METHODS: Available data on UK noncommercial trials (were obtained from the two relevant registries: ISRCTN register for the UK, and US ClinicalTrials.gov. Data on each trial were sorted by start year, and compared with the: 2003 survey, and UKCRN portfolio database from 2007. RESULTS: The number of UK noncommercial trials registered rose from 25 in 1990 to 188 in 1999, peaked at 533 in 2003, and fell back to 334 in 2009. Total trials registered was similar to but slightly above those in the 2003 survey up to 1998, then rose sharply to 2002 before falling to 2007. From 2007 to 2009 the number registered to start each year was similar to but slightly above the UKCRN database. Less than 10% of UK noncommercial trials registered with ClinGov for most years before 2005, but this rose to 35% by 2009. CONCLUSIONS: For the periods of overlap, trial registration data provide fairly similar totals to other sources on the number of noncommercial trials starting each year. The rise and fall in the number of trials registered between 1999 and 2007 was due to those registered in the ISRCTN database as funded by NHS Trusts. After 2007, the number of trials registered as funded by NHS Trusts has fallen in the ISRCTN register but these trials may have migrated to the US ClinGov register. The total number of noncommercial trial starts, excluding those funded by NHS Trusts, has been upward since around 2002. By 2009 the two main funders were NIHR and charities. Feasibility of using registration data to monitor the number of noncommercial trials has been demonstrated but is complicated by the use of two registers and difficulties in accessing the data. We recommend an annual report on the number of noncommercial trials registering each year.

Potential benefits of using a toolkit developed to aid in the adaptation of HTA reports: a case study considering positron emission tomography (PET) and Hodgkin's disease.

BACKGROUND: The preparation of HTA reports requires a great deal of time, effort and resource, and there is a desire to improve efficiency, avoid duplication of effort and facilitate the transfer of knowledge between countries. This is of particular importance for countries with more limited resources which have less capacity to produce their own reports. The aim of this study was to investigate the extent of duplication of published Health Technology Assessment (HTA) reports, on the same technology, for the same indication; using positron emission tomography (PET) for lung cancer and Hodgkin's disease as a case study. This was done in order to assess the potential usefulness of a toolkit developed to aid in the adaptation of HTA reports from one context or country to another. METHODS: A systematic search of the National Institute for Health Research (NIHR) CRD HTA database was conducted in June 2008 in order to identify full HTA reports containing information on the use of PET for lung cancer and Hodgkin's disease, written in English, and readily available on the web. The contents of the reports identified were then examined to assess the extent of duplication of content between reports and potential for the use of the toolkit. RESULTS: From 132 records of HTA reports about PET, 8 reports were identified as fulfilling all the criteria set, and therefore demonstrating potential duplication of effort. All these reports covered four similar domains, technology use, safety, effectiveness and economic evaluation. Five of the reports also considered organisational aspects. CONCLUSIONS: There was some duplication of effort in the preparation of HTA reports concerned with the use of PET for lung cancer and Hodgkin's disease. This is an example of where resource could have been conserved and time saved by the use of a toolkit developed to aid in the adaptation of HTA reports from one context to another.

The adaptation of health technology assessment reports: identification of the need for, and development of, a toolkit to aid the process.

OBJECTIVES: Europe has many health technology assessment (HTA) agencies, each producing their own HTA reports. Adapting HTA reports for different contexts could reduce the need for multiple reports on the same health technology with resultant saving of time and resources. This study aims to examine and understand the process of adaptation, and to develop a toolkit that would help the adaptation of reports produced by other countries. METHODS: The methods used were a review of the literature; a survey of twenty-nine European HTA organizations, two rounds of a Delphi survey, a face-to-face meeting of twenty-one European network for Health Technology Assessment (EUnetHTA) representatives, iterative rounds of review, and two rounds of quality assurance testing (termed applicability testing). RESULTS: Descriptions of previous examples of adaptation in the literature are sparse. Most respondents had previous experience in adapting reports, and all believed that adaptation was useful, and there was the ability to benefit from the use of a toolkit to aid in the process. EUnetHTA Partners developed and tested an adaptation toolkit. The toolkit is composed of a series of checklists and resources that identify or clarify the relevance, reliability, and transferability of data and information from existing reports. CONCLUSIONS: Consensus of opinion from twenty-nine European organizations/networks has indicated that the adaptation of HTA reports would be desirable and beneficial. A toolkit was developed to help with the adaptation of HTA reports produced in other settings. This collection of resources is available for use by all HTA agencies and can be accessed at: http://www.eunethta.net/upload/WP5/EUnetHTA_HTA_Adaptation_Toolkit_October08.pdf.

The health technology assessment adaptation toolkit: description and use.

OBJECTIVES: Adapting health technology assessment (HTA) reports for different contexts could reduce the need for multiple reports on the same health technology with resultant saving of time and resources. This article describes an instrument, the adaptation toolkit, which has been developed to aid in the process of adaptation of HTA reports. METHODS: The toolkit was developed by a partnership of HTA agencies and networks from across Europe. The role of the toolkit is to guide the user through the process of selecting possible relevant material from these report(s), assessing the relevance, reliability, and transferability of the material, and adapting it for the desired context. RESULTS: The adaptation toolkit has been developed, it comprises a collection of resources that help the user assess whether data and information in existing HTA reports should and could be adapted for their own setting. The toolkit contains two sections: a preliminary speedy sifting section and the main toolkit. The main toolkit includes five domains: (i) technology use and development, (ii) safety, (iii) effectiveness (including efficacy), (iv) economic evaluation, and (v) organizational aspects. Legal, ethical, and social aspects are beyond the scope of the toolkit. The toolkit is designed for the adaptation of evidence synthesis rather than primary research. CONCLUSIONS: The completed current version of the toolkit contains checklists and resources to aid in the adaptation of HTA reports. This collection of resources is available for use by all HTA agencies and can be accessed at: http://www.eunethta.net/upload/WP5/EUnetHTA_HTA_Adaptation_Toolkit_October08.pdf..

Enhancing understanding: the development of a glossary of health technology assessment adaptation terms.

BACKGROUND: The way people use health technology assessment (HTA) terms varies considerably across Europe. Such variation can lead to misunderstandings when reading HTA reports from different contexts. This work is one of the outputs of the EUnetHTA Project and was undertaken between 2006 and 2008. OBJECTIVES: The aim of this study was to develop a glossary of HTA adaptation terms to help reduce the misunderstandings of terms used in HTA reports from contexts other than the reader's own. METHODS: Several HTA glossaries were examined to identify ways in which an additional glossary could offer readers something new and to identify adaptation terms for inclusion. Twenty-eight European HTA organizations provided terms for the glossary and drafted descriptions and examples of how each specific term was used in their particular setting. The organizations then commented on the descriptions provided by the other groups and worked together to draft a single description for certain terms. RESULTS: A glossary of HTA adaptation terms was developed. It provides a comprehensive range of descriptions, examples, and comments for forty-two potentially confusing HTA terms related to adaptation. CONCLUSIONS: This glossary will be a valuable resource for European HTA agencies when reading HTA reports produced in different contexts and for adapting HTA reports produced in other countries. The glossary will help improve understanding and help facilitate the adaptation process.

European network for health technology assessment, EUnetHTA: planning, development, and implementation of a sustainable European network for health technology assessment.

OBJECTIVES: The European network on Health Technology Assessment (EUnetHTA) aimed to produce tangible and practical results to be used in the various phases of health technology assessment and to establish a framework and processes to support this. This article presents the background, objectives, and organization of EUnetHTA, which involved a total of sixty-four partner organizations. METHODS: Establishing an effective and sustainable structure for a transnational network involved many managerial, policy, and methodological tools, according to the objective of each task or Work Package. Transparency in organization, financial transactions, and decision making was a key principle in the management of the Project as was the commitment to appropriately involve stakeholders. RESULTS: EUnetHTA activities resulted in a clear management and governance structure, efficient partnership, and transnational cooperation. The Project developed a model for sustainable continuation of the EUnetHTA Collaboration. CONCLUSIONS: The EUnetHTA Project achieved its goals by producing a suite of practical tools, a strong network, and plans for continuing the work in a sustainable EUnetHTA Collaboration that facilitates and promotes the use of HTA at national and regional levels. Responsiveness to political developments in Europe should be balanced with maintaining a high level of ambition to promote independent, evidence-based information and well-tested tools for best practice based on a strong network of HTA institutions.

Carbon cost of pragmatic randomised controlled trials: retrospective analysis of sample of trials.

OBJECTIVE: To calculate the global warming potential, in carbon dioxide (CO(2)) equivalent emissions, from a sample of pragmatic randomised controlled trials. DESIGN: Retrospective analysis. Data source Internal data held by NIHR Evaluation, Trials and Studies Coordinating Centre. Studies included All eligible pragmatic randomised controlled trials funded by the NIHR Health Technology Assessment programme during 2002 and 2003. MAIN OUTCOME MEASURE: CO(2) equivalents for trial activities calculated with standard conversion factors. RESULTS: 12 pragmatic randomised controlled trials involving more than 4800 participants and a wide range of technologies were included. The average CO(2) emission generated by the trials was 78.4 (range 42.1-112.7) tonnes. This is equivalent to that produced in one year by approximately nine people in the United Kingdom. Commuting to work by the trial team generated the most emissions (average 21 (11.5-35.0) tonnes per trial), followed by study centres' fuel use (18 (9.3-32.2) tonnes per trial), trial team related travel (15 (2.0-29.0) tonnes per trial), and participant related travel (13 (0-46.7) tonnes per trial). CONCLUSIONS: CO(2) emissions from pragmatic randomised controlled trials are generated in areas where steps could be taken to reduce them. A large proportion of the CO(2) emissions come from travel related to various aspects of a trial. The results of this research are likely to underestimate the total CO(2) emissions associated with the trials studied, because of the sources of information available. Further research is needed to explore the additional CO(2) emissions generated by clinical trials, over and above those generated by routine care. The results from this project will feed into NIHR guidelines that will advise researchers on how to reduce CO(2) emissions.

A pilot Internet "value of health" panel: recruitment, participation and compliance.

OBJECTIVES: To pilot using a panel of members of the public to provide preference data via the Internet METHODS: A stratified random sample of members of the general public was recruited and familiarized with the standard gamble procedure using an Internet based tool. Health states were periodically presented in "sets" corresponding to different conditions, during the study. The following were described: Recruitment (proportion of people approached who were trained); Participation (a) the proportion of people trained who provided any preferences and (b) the proportion of panel members who contributed to each "set" of values; and Compliance (the proportion, per participant, of preference tasks which were completed). The influence of covariates on these outcomes was investigated using univariate and multivariate analyses. RESULTS: A panel of 112 people was recruited. 23% of those approached (n = 5,320) responded to the invitation, and 24% of respondents (n = 1,215) were willing to participate (net = 5.5%). However, eventual recruitment rates, following training, were low (2.1% of those approached). Recruitment from areas of high socioeconomic deprivation and among ethnic minority communities was low. Eighteen sets of health state descriptions were considered over 14 months. 74% of panel members carried out at least one valuation task. People from areas of higher socioeconomic deprivation and unmarried people were less likely to participate. An average of 41% of panel members expressed preferences on each set of descriptions. Compliance ranged from 3% to 100%. CONCLUSION: It is feasible to establish a panel of members of the general public to express preferences on a wide range of health state descriptions using the Internet, although differential recruitment and attrition are important challenges. Particular attention to recruitment and retention in areas of high socioeconomic deprivation and among ethnic minority communities is necessary. Nevertheless, the panel approach to preference measurement using the Internet offers the potential to provide specific utility data in a responsive manner for use in economic evaluations and to address some of the outstanding methodological uncertainties in this field.

What value health?: A review of health state values used in early technology assessments for NICE.

The objective of this article was to review the methods used to obtain quality-of-life (utility) weights reported in assessments carried out for the National Institute for Health and Clinical Excellence (NICE).The design of the review was a cross-sectional survey. Health technology assessment (HTA) reports published on the NICE website up to May 2003 were reviewed. Data were extracted on the following: the approach to utility estimation (direct or indirect), how health states were described for indirect estimation, valuation techniques used (standard gamble [SG], time trade-off [TTO], visual analogue scale [VAS], etc.), whether uncertainty in utility estimates was explored in cost-utility analyses, and whether utility values were identified as a priority for further research by assessment authors.Fifty-six assessments were reviewed, of which 28 reported 45 cost-utility analyses. There was striking variation in the values used to describe different health states. Data from patients were used in 15 (33%) analyses, from the general public in 10 (22%) and from clinicians in 4 (9%). In 16 (36%) cases, the source for utility estimates was unclear. Health states were described using a range of generic and disease-specific measures, although the EQ-5D was used most frequently. In 25 analyses (56%), the valuation technique used was not reported. TTO was used in 11 (24%), SG in 3 (7%), magnitude estimation in 5 (11%) and VAS in 1 (2%). Sensitivity analyses based on utility values were reported in 25 cases (56%), more commonly in reports of analyses carried out by independent teams than technology sponsors although this may be subject to reporting bias. Further research into quality of life was recommended in 17 (61%) of the 28 assessment reports that contained at least one cost-utility analysis. Greater transparency and consistency are required in reporting the methods used to obtain quality-of-life weights in cost-utility analyses, and better sources of data are required. Methodological variation results in important differences in values. Therefore, caution must be exercised when comparing the results of different cost-utility analyses.

Health technology assessment in England and Wales.

In this study, we describe the current state of the Health Technology Assessment (HTA) system in England and Wales. This system rests on a distinction between assessment and appraisal and has three main strands: researcher-led HTA, the research and development program, and the HTA-NICE (National Institute for Clinical Excellence) process. We outline the pressures for HTA and how it has evolved in the British National Health Service. We discuss how HTA priorities are chosen, how HTA information is collected and assessed, how HTA evidence is used, and we make some observations about its impact. In our discussion, we consider some limitations of the HTA system, its possible divergence from evidence-based health care, its centralization, and some of the key challenges for managing HTA-driven policy. But we remain hopeful that HTA can contribute to better and more explicit decision-making within England and Wales.

Health technology assessment: history and demand.

Health technology assessment (HTA)--the provision to decision makers of information on the value of treatments and tests--has come of age in the last two decades. But it has deep roots in health care, with notable landmarks in (1) the mid eighteenth-century development of empiricism, (2) the twentieth century interest in outcomes and variation in health care, and (3) the pioneering work of Archie Cochrane and others in the 1970s. Three main forces have driven the recent developments of HTA: a combination of concerns about the adoption of unproven technologies, rising costs, and an inexorable rise in consumer expectations. The HTA response, essentially initiatives supporting the provision of reliable synthesised research information on the effects and costs of health technologies, have been well supported in the United Kingdom and internationally. We can be sure that HTA is here to stay.

HTA responses and the classic HTA report.

Reports produced in response to the need for health technology assessment (HTA) vary greatly in the methods they use, depending on the decision-maker's needs, the technology's characteristics and the resources available. HTA reports vary from the brief, such as 'vignettes' produced when a new technology emerges, to the exhaustive, such as 'Cochrane reviews' synthesising a mature evidence base. They may address a wide range of different questions. 'Classic HTAs', typically those reports prepared to support NICE appraisal decisions, seek to use scientifically rigorous methods to address a focused policy question. These are based on systematic reviews of the effectiveness evidence but this is often fed into economic models, in order to generate estimates of cost-effectiveness. Future developments in HTA responses will reflect both the embedding of systematic methods and a growing responsiveness to customer needs.