Impact of NIHR HTA Programme funded research on NICE clinical guidelines: a retrospective cohort.

BACKGROUND: It is vitally important that there is a connection between health research and clinical practice. Indications as to the impact of the research on evidence-based practice and policy can be obtained by tracking the use of outputs of health research, especially its use in clinical guidelines (CGs). This study aims to assess the proportion of National Institute for Health and Care Excellence (NICE) CGs citing National Institute for Health Research Health Technology Assessment (NIHR HTA) studies and the impact of evidence from those studies on the included NICE CGs. METHODS: This is a retrospective cohort study assessing the proportion of NICE CGs from all NICE CGs issued between April 2001 and April 2012, which cited evidence from studies funded by the NIHR HTA Programme and the impact of those studies on the CGs as the primary and secondary outcome measures. RESULTS: Of the cohort of NICE CGs (n = 122), 3 (2%) CGs were based on previous NIHR HTA reports and would not have been issued in that form without those NIHR HTA studies, 90 (74%) included evidence from NIHR HTA studies, and 29 (24%) did not include evidence from NIHR HTA studies. The impact of NIHR HTA evidence on NICE CGs varied in the type and quantity of data used. CONCLUSIONS: Findings suggest that NIHR HTA funded research impacts on clinical guidance from NICE and hence is well connected to both clinical practice and policy.

Features of successful bids for funding of applied health research: a cohort study.

BACKGROUND: The literature suggests that research funding decisions may be influenced by criteria such as gender or institution of the principal investigator (PI). The aim of this study was to investigate the association between characteristics of funding applications and success when considered by a research funding board. METHODS: We selected a retrospective cohort of 296 outline applications for primary research (mainly pragmatic clinical trials) submitted to the commissioning board of the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme between January 1st 2006 and December 31st 2009. We selected proposals submitted to the commissioned NIHR HTA work stream as they addressed issues which the programme already deemed to be important, hence the priority of the research question was not considered as one of the selection criteria for success or failure. Main outcome measures were success or failure at short-listing and in obtaining research funding. RESULTS: The characteristics of applications associated with success at shortlisting and funding were multi-disciplinarity of the team (OR 19.94 [5.13, 77.50], P < 0.0001), particularly inclusion of a statistician (OR 3.76 [2.21, 6.37], P < 0.0001), and the completion of a pilot/feasibility study (OR 4.11 [1.24, 13.62], P = 0.0209). The gender of the PI was not associated with success or failure at either stage. The PI's affiliation institution was not associated with success or failure at shortlisting. CONCLUSIONS: The gender of the PI was not associated with success or failure. The characteristics of research applications most strongly associated with success were related to the range of expertise in the team and the completion of a pilot or feasibility study.

Developing a checklist for research proposals to help describe health service interventions in UK research programmes: a mixed methods study.

BACKGROUND: One of the most common reasons for rejecting research proposals in the National Institute for Health Research (NIHR) Health Services and Delivery Research (HS&DR) Programme is the failure to adequately specify the intervention or context in research proposals. Examples of failed research proposals include projects to assess integrated care models, use of generic caseworkers, or new specialist nurse services. These are all important service developments which need evaluation, but the lack of clarity about the intervention and context prevented these research proposals from obtaining funding. The purpose of the research presented herein was to develop a checklist, with key service intervention and contextual features, for use by applicants to the NIHR HS&DR Programme to potentially enhance the quality of research proposals. METHODS: The study used mixed methods to identify the need for and develop and test a checklist. Firstly, this included assessing existing checklists in peer-reviewed literature relevant to organisational health research. Building on existing work, a new checklist was piloted. Two reviewers used a small sample (n = 16) of research proposals to independently assess the relevance of the checklist to the proposal and the degree of overlap or gaps between the constructs. The next two stages externally validated the revised checklist by collecting qualitative feedback from researchers and experts in the field. RESULTS: The initial checklist was developed from existing checklists which included domains of intervention and context. The constructs and background to each were developed through review of existing literature. Eight researchers provided feedback on the checklist, which was generally positive. This iterative process resulted in changes to the checklist, collapsing two constructs and providing more prompts for others; the final checklist includes six constructs. CONCLUSIONS: Features relating to intervention and context should be well described to increase the quality of research proposals and enhance the chances of the research receiving funding. Existing checklists do not have enough focus on areas relevant to research proposals in complex health service interventions, such as workforce. A formative checklist has been developed, and tested by end users. Tentative findings suggest usefulness and acceptability of such a tool but further work is needed for full validation.

Potential benefits of using a toolkit developed to aid in the adaptation of HTA reports: a case study considering positron emission tomography (PET) and Hodgkin's disease.

BACKGROUND: The preparation of HTA reports requires a great deal of time, effort and resource, and there is a desire to improve efficiency, avoid duplication of effort and facilitate the transfer of knowledge between countries. This is of particular importance for countries with more limited resources which have less capacity to produce their own reports. The aim of this study was to investigate the extent of duplication of published Health Technology Assessment (HTA) reports, on the same technology, for the same indication; using positron emission tomography (PET) for lung cancer and Hodgkin's disease as a case study. This was done in order to assess the potential usefulness of a toolkit developed to aid in the adaptation of HTA reports from one context or country to another. METHODS: A systematic search of the National Institute for Health Research (NIHR) CRD HTA database was conducted in June 2008 in order to identify full HTA reports containing information on the use of PET for lung cancer and Hodgkin's disease, written in English, and readily available on the web. The contents of the reports identified were then examined to assess the extent of duplication of content between reports and potential for the use of the toolkit. RESULTS: From 132 records of HTA reports about PET, 8 reports were identified as fulfilling all the criteria set, and therefore demonstrating potential duplication of effort. All these reports covered four similar domains, technology use, safety, effectiveness and economic evaluation. Five of the reports also considered organisational aspects. CONCLUSIONS: There was some duplication of effort in the preparation of HTA reports concerned with the use of PET for lung cancer and Hodgkin's disease. This is an example of where resource could have been conserved and time saved by the use of a toolkit developed to aid in the adaptation of HTA reports from one context to another.

The adaptation of health technology assessment reports: identification of the need for, and development of, a toolkit to aid the process.

OBJECTIVES: Europe has many health technology assessment (HTA) agencies, each producing their own HTA reports. Adapting HTA reports for different contexts could reduce the need for multiple reports on the same health technology with resultant saving of time and resources. This study aims to examine and understand the process of adaptation, and to develop a toolkit that would help the adaptation of reports produced by other countries. METHODS: The methods used were a review of the literature; a survey of twenty-nine European HTA organizations, two rounds of a Delphi survey, a face-to-face meeting of twenty-one European network for Health Technology Assessment (EUnetHTA) representatives, iterative rounds of review, and two rounds of quality assurance testing (termed applicability testing). RESULTS: Descriptions of previous examples of adaptation in the literature are sparse. Most respondents had previous experience in adapting reports, and all believed that adaptation was useful, and there was the ability to benefit from the use of a toolkit to aid in the process. EUnetHTA Partners developed and tested an adaptation toolkit. The toolkit is composed of a series of checklists and resources that identify or clarify the relevance, reliability, and transferability of data and information from existing reports. CONCLUSIONS: Consensus of opinion from twenty-nine European organizations/networks has indicated that the adaptation of HTA reports would be desirable and beneficial. A toolkit was developed to help with the adaptation of HTA reports produced in other settings. This collection of resources is available for use by all HTA agencies and can be accessed at: http://www.eunethta.net/upload/WP5/EUnetHTA_HTA_Adaptation_Toolkit_October08.pdf.

The health technology assessment adaptation toolkit: description and use.

OBJECTIVES: Adapting health technology assessment (HTA) reports for different contexts could reduce the need for multiple reports on the same health technology with resultant saving of time and resources. This article describes an instrument, the adaptation toolkit, which has been developed to aid in the process of adaptation of HTA reports. METHODS: The toolkit was developed by a partnership of HTA agencies and networks from across Europe. The role of the toolkit is to guide the user through the process of selecting possible relevant material from these report(s), assessing the relevance, reliability, and transferability of the material, and adapting it for the desired context. RESULTS: The adaptation toolkit has been developed, it comprises a collection of resources that help the user assess whether data and information in existing HTA reports should and could be adapted for their own setting. The toolkit contains two sections: a preliminary speedy sifting section and the main toolkit. The main toolkit includes five domains: (i) technology use and development, (ii) safety, (iii) effectiveness (including efficacy), (iv) economic evaluation, and (v) organizational aspects. Legal, ethical, and social aspects are beyond the scope of the toolkit. The toolkit is designed for the adaptation of evidence synthesis rather than primary research. CONCLUSIONS: The completed current version of the toolkit contains checklists and resources to aid in the adaptation of HTA reports. This collection of resources is available for use by all HTA agencies and can be accessed at: http://www.eunethta.net/upload/WP5/EUnetHTA_HTA_Adaptation_Toolkit_October08.pdf..

Influence of external peer reviewer scores for funding applications on funding board decisions: a retrospective analysis of 1561 reviews.

OBJECTIVES: To evaluate the influence of external peer reviewer scores on the National Institute for Health Research (NIHR) research funding board decisions by the number of reviewers and type of reviewer expertise. DESIGN: Retrospective analysis of external peer review scores for shortlisted full applications for funding (280 funding applications, 1236 individual reviewers, 1561 review scores). SETTING: Four applied health research funding programmes of NIHR, UK. MAIN OUTCOME MEASURES: Board decision to fund or not fund research applications. RESULTS: The mean score of reviewers predicted funding decisions better than individual reviewer scores (area under the receiver operating characteristic (ROC) curve 0.75, 95% CI 0.69 to 0.81 compared with 0.62, CI 0.59 to 0.65). There was no substantial improvement in how accurately mean reviewer scores predicted funding decisions when the number of reviewers increased above 4 (area under ROC curve 0.75, CI 0.59 to 0.91 for four reviewers; 0.80, CI 0.67 to 0.92 for seven or more). Reviewers with differing expertise influenced the board's decision equally, including public and patient reviewers (area under ROC curves from 0.57, CI 0.47 to 0.66 for health economists to 0.64, CI 0.57 to 0.70 for subject-matter experts). The areas under the ROC curves were quite low when using reviewers' scores, confirming that boards do not rely solely on those scores alone to make their funding decisions, which are best predicted by the mean board score. CONCLUSIONS: Boards value scores that originate from a diverse pool of reviewers. On the basis of independent reviewer score alone, there is no detectable benefit of using more than four reviewer scores in terms of their influence on board decisions, so to improve efficiency, it may be possible to avoid using larger numbers of reviewers. The funding decision is best predicted by the board score.

Evaluation of stakeholder views on peer review of NIHR applications for funding: a qualitative study.

OBJECTIVES: Innovations resulting from research have both national and global impact, so selecting the most promising research studies to fund is crucial. Peer review of research funding applications is part of the selection process, and requires considerable resources. This study aimed to elicit stakeholder opinions about which factors contribute to and influence effective peer review of funding applications to the UK National Institute for Health Research (NIHR), and to identify possible minor improvements to current processes and any major changes or potential innovations to achieve a more efficient peer review process. DESIGN: Qualitative interviews with 30 stakeholders involved in the peer review process. PARTICIPANTS: Participants were drawn from three NIHR coordinating centres and represented four types of stakeholders: board members with responsibility for making funding decisions, applicants, external peer reviewers and NIHR staff. METHODS: All interviews were conducted by telephone apart from three that were face to face with NIHR staff. Data were analysed using a thematic template method. RESULTS: The responses from NIHR staff, board members and reviewers differed from those received from applicants. The first three groups focused on how well the process of peer review did or did not function. The applicants mentioned these points but in addition often reflected on how their personal application was assessed. Process improvements suggested included: developing a more proportionate review process; providing greater guidance, feedback, training, acknowledgement or incentives for peer reviewers; reducing the time commitment and amount of paperwork; and asking reviewers to comment on the importance, strengths and weaknesses of applications and flaws which are potentially 'fixable'. CONCLUSIONS: Overall, participants were supportive of the need for peer review in evaluating applications for research funding. This study revealed which parts of the process are working well and are valued, and barriers, difficulties and potential areas for improvement and development.

Developing clinical rules to predict urinary tract infection in primary care settings: sensitivity and specificity of near patient tests (dipsticks) and clinical scores.

BACKGROUND: Suspected urinary tract infection (UTI) is one of the most common presentations in primary care. Systematic reviews have not documented any adequately powered studies in primary care that assess independent predictors of laboratory diagnosis. AIM: To estimate independent clinical and dipstick predictors of infection and to develop clinical decision rules. DESIGN OF STUDY: Validation study of clinical and dipstick findings compared with laboratory testing. SETTING: General practices in the south of England. METHOD: Laboratory diagnosis of 427 women with suspected UTI was assessed using European urinalysis guidelines. Independent clinical and dipstick predictors of diagnosis were estimated. RESULTS: UTI was confirmed in 62.5% of women with suspected UTI. Only nitrite, leucocyte esterase (+ or greater), and blood (haemolysed trace or greater) independently predicted diagnosis (adjusted odds ratios 6.36, 4.52, 2.23 respectively). A dipstick decision rule, based on having nitrite, or both leucocytes and blood, was moderately sensitive (77%) and specific (70%); positive predictive value (PPV) was 81% and negative predictive value (NPV) was 65%. Predictive values were improved by varying the cut-off point: NPV was 73% for all three dipstick results being negative, and PPV was 92% for having nitrite and either blood or leucocyte esterase. A clinical decision rule, based on having two of the following: urine cloudiness, offensive smell, and dysuria and/or nocturia of moderate severity, was less sensitive (65%) (specificity 69%; PPV 77%, NPV 54%). NPV was 71% for none of the four clinical features, and the PPV was 84% for three or more features. CONCLUSIONS: Simple decision rules could improve targeting of investigation and treatment. Strategies to use such rules need to take into account limited negative predictive value, which is lower than expected from previous research.

Cardiovascular implanted electronic devices in people towards the end of life, during cardiopulmonary resuscitation and after death: guidance from the Resuscitation Council (UK), British Cardiovascular Society and National Council for Palliative Care.

The Resuscitation Council (UK), the British Cardiovascular Society (including the British Heart Rhythm Society and the British Society for Heart Failure) and the National Council for Palliative Care recognise the importance of providing clear and consistent guidance on management of cardiovascular implanted electronic devices (CIEDs) towards the end of life, during cardiorespiratory arrest and after death. This document has been developed to provide guidance for the full range of healthcare professionals who may encounter people with CIEDs in the situations described and for healthcare managers and commissioners. The authors recognise that some patients and people close to patients may also wish to refer to this document. It is intended as an initial step to help to ensure that people who have CIEDs, or are considering implantation of one, receive explanation of and understand the practical implications and decisions that this entails; to promote a good standard of care and service provision for people in the UK with CIEDs in the circumstances described; to offer relevant ethical and legal guidance on this topic; to offer guidance on the delivery of services in relation to deactivation of CIEDs where appropriate; to offer guidance on whether any special measures are needed when a person with a CIED receives cardiopulmonary resuscitation; and to offer guidance on the actions needed when a person with a CIED dies.

Cost effectiveness of management strategies for urinary tract infections: results from randomised controlled trial.

OBJECTIVE: To assess the cost effectiveness of different management strategies for urinary tract infections. DESIGN: Cost effectiveness analysis alongside a randomised controlled trial with a one month follow-up. SETTING: Primary care. PARTICIPANTS: 309 non-pregnant adult women aged 18-70 presenting with suspected urinary tract infection. INTERVENTIONS: Patients were randomised to five basic management approaches: empirical antibiotics, empirical delayed (by 48 hours) antibiotics, or targeted antibiotics based on either a high symptom score (two or more of urine cloudiness, smell, nocturia, dysuria), dipstick results (nitrite or leucocytes and blood), or receipt of a positive result on midstream urine analysis. MAIN OUTCOME MEASURE: Duration of symptoms and cost of care. RESULTS: Management with targeted antibiotics with midstream urine analysis was more costly over the period of one month. Costs for the midstream urine analysis and dipstick management groups were pound37 and pound35, respectively; these compared with pound31 for immediate antibiotics. Cost effectiveness acceptability curves suggested that if avoiding a day of moderately bad symptoms was valued at less than pound10, then immediate antibiotics is likely to be the most cost effective strategy. For values over pound10, targeted antibiotics with dipstick testing becomes the most cost effective strategy, though because of the uncertainty we can never be more than 70% certain that this strategy truly is the most cost effective. CONCLUSION: Dipstick testing with targeted antibiotics is likely to be cost effective if the value of saving a day of moderately bad symptoms is pound10 or more, but caution is required given the considerable uncertainty surrounding the estimates.

Validating the prediction of lower urinary tract infection in primary care: sensitivity and specificity of urinary dipsticks and clinical scores in women.

BACKGROUND: Dipsticks are one of the most commonly used near-patient tests in primary care, but few clinical or dipstick algorithms have been rigorously developed. AIM: To confirm whether previously documented clinical and dipstick variables and algorithms predict laboratory diagnosis of urinary tract infection (UTI). DESIGN OF STUDY: Validation study. SETTING: Primary care. METHOD: A total of 434 adult females with suspected lower UTI had bacteriuria assessed using the European Urinalysis Guidelines. RESULTS: Sixty-six per cent of patients had confirmed UTI. The predictive values of nitrite, leucocyte esterase (+ or greater), and blood (haemolysed trace or greater) were confirmed (independent multivariate odds ratios = 5.6, 3.5, and 2.1 respectively). The previously developed dipstick rule--based on presence of nitrite, or both leucocytes and blood-- was moderately sensitive (75%) but less specific (66%; positive predictive value [PPV] 81%, negative predictive value [NPV] 57%). Predictive values were improved by varying the cut-off point: NPV was 76% for all three dipstick results being negative; the PPV was 92% for having nitrite and either blood or leucocyte esterase. Urine offensive smell was not found to be predictive in this sample; for a clinical score using the remaining three predictive clinical features (urine cloudiness, dysuria, and nocturia), NPV was 67% for none of the features, and PPV was 82% for three features. CONCLUSION: A clinical score is of limited value in increasing diagnostic precision. Dipstick results can modestly improve diagnostic precision but poorly rule out infection. Clinicians need strategies to take account of poor NPVs.

The journey from self-care to GP care: a qualitative interview study of women presenting with symptoms of urinary tract infection.

BACKGROUND: Urinary tract infection (UTI) is one of the commonest acute infections presenting to primary care. Little is known of women's experiences of UTI; self-care strategies and key triggers for their consulting behaviour are also little known. AIM: To explore women's experiences of self-care and their journey to GP care, when faced with symptoms of a UTI. DESIGN OF STUDY: Qualitative semi-structured interview study with women recruited to a larger UK trial of different management strategies for UTI. SETTING: General practices across four counties in southern England. METHOD: Twenty-one women were interviewed about the experiences they had prior to their GP visit, self-care strategies, and triggers for help seeking. Interviews were analysed thematically, using principles of analytic induction. RESULTS: Women reported a process of evaluation, monitoring, re-evaluation, and, finally, consulting in order to meet their needs. Four key triggers for consulting were identified: failure to alleviate symptoms through self-care; symptom duration and escalation; impeding normal functioning and the fulfilment of social roles; and concern that it may be or become a serious illness. CONCLUSION: Although UTI is often self-limiting, when taking patient histories and formulating their management strategies clinicians need to take into account women's often painful experience, their efforts to resolve symptoms prior to consulting, and their fears that the symptoms may indicate something more serious than a UTI.