Transferability of Lessons From Program Evaluations: Iron Laws, Hiding Hands and the Evidence Ecosystem.

Assessing the transferability of lessons from social research or evaluation continues to raise challenges. Efforts to identify transferable lessons can be based on two different forms of argumentation. The first draws upon statistics and causal inferences. The second involves constructing a reasoned case based on weighing up different data collected along the causal chain from designing to delivery. Both approaches benefit from designing research based upon existing evidence and ensuring that the descriptions of the programme, context, and intended beneficiaries are sufficiently rich. Identifying transferable lessons should not be thought of as a one-off event but involves contributing to the iterative and learning of a scientific community. To understand the circumstances under which findings can be confidently transferred, we need to understand: (1) How far and why outcomes of interest have multiple, interacting and fluctuating causes. (2) The program design and implementation capacity. (3) Prior knowledge and causal landscapes (and how far these are included in the theory of change). (4) New and relevant knowledge; what can we learn in our 'disputatious community of truth seekers'.

COVID-19 Genomics UK (COG-UK) Consortium: Final Report.

The ability to sequence and understand different variants of the SARS-CoV-2 virus and their impact is crucial to inform policy and public health decisions. Soon after the UK went into its first lockdown in March 2020, the CCOVID-19 Genomics UK (COG-UK) Consortium was launched. COG-UK is a collaboration of experts in pathogen genomics including academic institutions, public health agencies, the Wellcome Sanger Institute, NHS Trusts and Lighthouse Labs. RAND Europe evaluated how COG-UK delivered against its objectives, for example how it contributed to advancing scientific knowledge about SARS-CoV-2, informing public health decisions, and providing information that can be used to evaluate the effectiveness of vaccines and treatments. The evaluation also examined the diverse factors that influenced COG-UK progress and impact, including enablers and challenges, and considered implications for the future.

Integrated Care in England - what can we Learn from a Decade of National Pilot Programmes?

INTRODUCTION: For more than a decade the English NHS has pursued integrated care through three national pilot programmes. The independent evaluators of these programmes here identify several common themes that inform the development of integrated care. DESCRIPTION: The three pilot programmes shared the aim of better coordination between hospital and community-based health services and between health and social care. Each programme recruited local pilot sites that designed specific interventions to support inter-professional and inter-organisational collaboration.The pilots were highly heterogenous and results varied both within and between the three programmes. While staff were generally positive about their achievements, pilots had mixed success especially in reducing unplanned hospital admissions. Common facilitators to achieving pilots' objectives included effective senior leadership and shared values, simple interventions and additional funding. Barriers included short timescales, poor professional engagement, information and data sharing problems, and conflicts with changing national policy. DISCUSSION: There was little stable or shared understanding of what 'integrated care' meant resulting in different practices and priorities. An increasing focus on reducing unplanned hospital use among national sponsors created a mismatch in expectations between local and national actors. CONCLUSION: Pilots in all three national programmes made some headway against their objectives but were limited in their impact on unplanned hospital admissions.

Transforming Urgent and Emergency Care and the Vanguard Initiative: Learning from Evaluation of the Southern Cluster.

Urgent and Emergency Care (UEC) vanguards aim to improve the quality, efficiency and effectiveness of UEC services so that patients receive the most appropriate care at the right time and in the right place, and so that unnecessary admissions to accident and emergency (A&E) and hospitals are reduced. The Southern Cluster comprises three such UEC vanguards. RAND Europe's evaluation examined the impacts of the vanguards, the processes underpinning delivery (and associated enablers and challenges), and implications for future policy and practice. The evaluation used a multi-method approach, including theories of change, document review, workshops, interviews, surveys and data dashboards. The Southern Cluster UEC vanguards have made progress across core activities. Clinical hubs are operational across the sites. Direct booking capacity into primary care is progressing more with out-of-hours than with in-hours services. Gradual but variable progress has been made towards joint planning and governance of UEC services. Efforts to ensure seamless data sharing between providers, and interoperable IT infrastructure are progressing somewhat slower than originally hoped. Vanguard funding, committed leadership and practical mechanisms to support joint working helped drive progress. Public engagement, workforce-capacity and data interoperability will need to be addressed for longer-term impact at scale. The study makes recommendations concerning: (i) UEC health and care workforce capacity-building, (ii) local-national coordination around UEC transformation, (iii) collaboration across localities and professions, (iv) support for an end-to-end UEC pathway with mutually reinforcing activities, (v) cost and outcome data, (vi) an interoperable data infrastructure, and (vii) capacity for evaluation and learning.

Innovation as a Driver of Quality and Productivity in UK Healthcare: Creating and Connecting Receptive Places-Emerging Insights.

The demand for health services in England is both growing and changing in nature, yet resources are limited in their ability to respond to the scale and scope of need. As a result, the NHS is under increasing pressures to realise productivity gains, while continuing to deliver high quality care. RAND Europe and the University of Manchester have been commissioned to conduct a study to examine the potential of innovation to respond to the challenges the NHS faces, and to help deliver value for money, efficient and effective services. "Innovation" in this study refers to any product, technology or service that is new to the NHS, or applied in a new way, aimed at delivering affordable and improved care. The learning we have gained adds considerable depth to the practical discussions presented regarding how innovation can be first nurtured and then made meaningful and actionable in a variety of settings. This is important given the complexity of health innovation systems and the diversity of elements that need to interact and work together for the overall system to function effectively. We share insights related to skills, capabilities and leadership; motivations and accountabilities; information and evidence; relationships and networks; patient and public engagement; and funding and commissioning. We will develop these detailed learning points into a more systematic analysis as the research evolves. The research is funded by the Department of Health Policy Research Programme, in close collaboration with NHS England and the Office of Life Sciences.

Evaluating the Role and Contribution of Innovation to Health and Wealth in the UK: A Review of Innovation, Health and Wealth: Phase 1 Final Report.

The Department of Health's Innovation, Health and Wealth (IHW) strategy aimed to introduce a more strategic approach to the spread of innovation across the NHS. This study represents the first phase of a three-year evaluation and aims to map progress towards the IHW strategy and its component actions. This evaluation used a combination of quantitative and qualitative methods: document review, key informant interviews and stakeholder survey. This study also forms the basis for selecting case studies for phase two of the evaluation. Our findings from the interviews and survey suggest broad stakeholder support for the overarching ambitions of the IHW strategy. However, we found variable progress towards the overarching objectives of the eight IHW themes and an ambiguous relationship between many of the themes' objectives and their actions. It was difficult to assess progress on IHW's actions as commitment to the actions, implementation guidance and expected outcomes of the actions were not clearly articulated. The Academic Health Science Networks (AHSNs) and the Small Business Research Initiative (SBRI) were reported to be working well, which may be attributed to their clear structures of accountability and earmarked budgets. However, survey respondents and interviewees raised concerns that budgetary pressures may limit the impact of both AHSNs and the SBRI. The main challenges identified for ongoing action were the resources available for their implementation (e.g. Medtech Briefings), lack of awareness of the initiative (e.g. the NICE Implementation Collaborative) and the design of the actions (e.g. the Innovation Scorecard, web portal and High Impact Innovations).

Galvanising the NHS to Adopt Innovation: The Feasibility and Practicality of Recommendations from the Interim Report of the Accelerated Access Review.

The Department of Health and the Wellcome Trust, in co-operation with NHS England, asked RAND Europe to conduct a limited consultation with key stakeholders about the practicality of measures and incentives proposed as part of the NHS Accelerated Access Review (AAR), which aims to assess the pathways for the development, assessment, and adoption of innovative medicines and medical technology. Through a focused engagement exercise with key healthcare stakeholders this project explored the implications of selected interim AAR propositions and feasibility of implementation for key actors, in primary and secondary care as well as commissioners and academia. Specifically, the project investigated the feasibility of implementation of three specific propositions including: a new earmarked fund to encourage AHSNs and other key innovation actors to re-design systems to embrace innovation; mobilising the influence of clinical system leaders to champion change; and encouraging secondary care organisations to take on "innovation champion" roles linked to financial incentives and a new emphasis on accountable care organisations. Data was collected on the feasibility of the three AAR propositions from a workshop with AHSN CEOs and Commercial Directors and interviews with senior NHS staff in three AHSN regions (South West, University College London Partners, and North East, North Cumbria). The study concludes with reflections on the feasibility of each recommendation and identifies factors expected to facilitate or challenge their implementation, as well as considering the wider cross cutting issues that may influence the adoption and diffusion of innovation in the NHS.

Community Hospitals in Selected High Income Countries: A Scoping Review of Approaches and Models.

BACKGROUND: There is no single definition of a community hospital in the UK, despite its long history. We sought to understand the nature and scope of service provision in community hospitals, within the UK and other high-income countries. METHODS: We undertook a scoping review of literature on community hospitals published from 2005 to 2014. Data were extracted on features of the hospital model and the services provided, with results presented as a narrative synthesis. RESULTS: 75 studies were included from ten countries. Community hospitals provide a wide range of services, with wide diversity of provision appearing to reflect local needs. Community hospitals are staffed by a mixture of general practitioners (GPs), nurses, allied health professionals and healthcare assistants. We found many examples of collaborative working arrangements between community hospitals and other health care organisations, including colocation of services, shared workforce with primary care and close collaboration with acute specialists. CONCLUSIONS: Community hospitals are able to provide a diverse range of services, responding to geographical and health system contexts. Their collaborative nature may be particularly important in the design of future models of care delivery, where emphasis is placed on integration of care with a key focus on patient-centred care.

Corporate responsibility for childhood physical activity promotion in the UK.

The alarming epidemic of obesity and physical inactivity at paediatric age urges societies to rise to the challenge of ensuring an active lifestyle. As one response to this, business enterprises are increasingly engaged in promoting sport and physical activity (PA) initiatives within the frame of corporate social responsibility (CSR). However, comparative analyses among industry sectors of CSR strategies for PA promotion with a particular focus on children are still lacking. This study aimed to explore (i) what are the CSR strategies for PA promotion adopted in different industry sectors and (ii) whether corporate engagement in promoting PA for children is supportive of children's rights to play and be physically active. Corporate pledges pertaining to CSR initiatives to promote PA were analysed. The hypothesis was that companies from different sectors employ different CSR strategies and that companies with a higher profile as regard to public health concerns for children tend to legitimate their action by adopting a compensatory strategy. Results show that the issue of PA promotion is largely represented within CSR commitments. CSR strategies for PA promotion vary across industry sectors and the adoption of a compensatory strategy for rising childhood obesity allows only a limited exploitation of the potential of CSR commitments for the provision of children's rights to play and be physically active. Actors within the fields of public health ethics, human rights and CSR should be considered complementary to develop mainstreaming strategies and improve monitoring systems of PA promotion in children.

Public acceptability of government intervention to change health-related behaviours: a systematic review and narrative synthesis.

BACKGROUND: Governments can intervene to change health-related behaviours using various measures but are sensitive to public attitudes towards such interventions. This review describes public attitudes towards a range of policy interventions aimed at changing tobacco and alcohol use, diet, and physical activity, and the extent to which these attitudes vary with characteristics of (a) the targeted behaviour (b) the intervention and (c) the respondents. METHODS: We searched electronic databases and conducted a narrative synthesis of empirical studies that reported public attitudes in Europe, North America, Australia and New Zealand towards interventions relating to tobacco, alcohol, diet and physical activity. Two hundred studies met the inclusion criteria. RESULTS: Over half the studies (105/200, 53%) were conducted in North America, with the most common interventions relating to tobacco control (110/200, 55%), followed by alcohol (42/200, 21%), diet-related interventions (18/200, 9%), interventions targeting both diet and physical activity (18/200, 9%), and physical activity alone (3/200, 2%). Most studies used survey-based methods (160/200, 80%), and only ten used experimental designs. Acceptability varied as a function of: (a) the targeted behaviour, with more support observed for smoking-related interventions; (b) the type of intervention, with less intrusive interventions, those already implemented, and those targeting children and young people attracting most support; and (c) the characteristics of respondents, with support being highest in those not engaging in the targeted behaviour, and with women and older respondents being more likely to endorse more restrictive measures. CONCLUSIONS: Public acceptability of government interventions to change behaviour is greatest for the least intrusive interventions, which are often the least effective, and for interventions targeting the behaviour of others, rather than the respondent him or herself. Experimental studies are needed to assess how the presentation of the problem and the benefits of intervention might increase acceptability for those interventions which are more effective but currently less acceptable.

The European Alcohol and Health Forum: First Monitoring Progress Report.

The European Alcohol and Health Forum was launched in June 2007 following the adoption by the European Commission in October 2006 of the EU strategy to reduce alcohol-related harm. It began as a stakeholder platform of 50 founding members from production and sales organisations, media and advertising organisations, NGOs that work to limit alcohol-related harm, research organisations, professional bodies and others. Membership of the Forum is voluntary and members are expected to commit formally and publicly to concrete actions to reduce alcohol-related harm. These concrete actions are referred to as "commitments." Each commitment must be put in writing in a standardised commitment form, which also includes monitoring information. Such information includes details of the objectives of the commitments, resources allocated to them and outputs produced, as well as dissemination of the results of the commitments alongside other information.

Changing the Translational Research Landscape: A Review of the Impacts of Biomedical Research Units in England.

This article describes a review of the Biomedical Research Units (BRU) scheme, undertaken for the Department of Health. This review was a perceptions audit of senior executives involved in the scheme, and explored what impact they felt the scheme is having on the translational research landscape. More specifically, we investigated whether and how institutional relationships between NHS and academic partners, industry and other health research system players are changing because of the scheme; how the scheme is helping build critical mass in specific priority disease areas; and the effects of any changes on efforts to deliver the broader goals set out in Best Research for Best Health. The views presented are those of study informants only. The information obtained through our interviews suggests that the BRU scheme is significantly helping shape the health research system to pursue translational research and innovation, with the clear goal of realising patient benefit. The BRUs are already contributing to observable changes in institutional relationships between the NHS and academic partners: trusts and medical schools are collaborating more closely than in the past, have signed up to the same vision of translational research from bench to bedside, and are managing and governing targeted research resources more professionally and transparently than in the past. There is also a stronger emphasis on engaging industry and more strategic thinking about strengthening regional and national collaboration with other hospital trusts, PCTs, research organisations, networks and development agencies. The scheme is also transforming capacity building in the health research system. This includes (i) developing and modernising facilities and equipment for translation; (ii) building a critical mass of human resources through recruitment and training, as well as improving retention of existing expertise; and (iii) helping ensure a steady flow of funds needed to sustain research activity and accelerate movement through the innovation pipeline. A number of centres are also trying to recreate the BRU model in new disease areas, with their own resources.

Sustainable Development in the National Health Service (NHS): The Views and Values of NHS Leaders.

This article presents National Health Service (NHS) leaders' views of priorities and approaches regarding sustainable development in the NHS. It was produced in close collaboration with the United Kingdom (UK) NHS Sustainable Development Unit (SDU), and it represents the first systematic picture of leadership views in the NHS. It also provides a commentary on ways forward. Analysis draws on results of a survey of 172 leaders of NHS organisations (primarily chief executives), 12 follow-up interviews, interviews with the SDU, and additional data and literature searches. A major conclusion is that almost all leaders consider sustainable development to be important for the NHS and that a focus on sustainability can most likely be aligned with delivering other corporate goals. Aligned incentives at all organisational levels and support for diversity are considered necessary to achieve sustainability, as well as relevant performance metrics. The main barrier is organisational culture.

Barriers and facilitators to integrating care: experiences from the English Integrated Care Pilots.

BACKGROUND: In 2008, the English Department of Health appointed 16 'Integrated Care Pilots' which used a range of approaches to provide better integrated care. We report qualitative analyses from a three-year multi-method evaluation to identify barriers and facilitators to successful integration of care. THEORY AND METHODS: Data were analysed from transcripts of 213 in-depth staff interviews, and from semi-structured questionnaires (the 'Living Document') completed by staff in pilot sites at six points over a two-year period. Emerging findings were therefore built from 'bottom up' and grounded in the data. However, we were then interested in how these findings compared and contrasted with more generic analyses. Therefore after our analyses were complete we then systematically compared and contrasted the findings with the analysis of barriers and facilitators to quality improvement identified in a systematic review by Kaplan et al. (2010) and the analysis of more micro-level shapers of behaviour found in Normalisation Process Theory (May et al. 2007). Neither of these approaches claims to be full blown theories but both claim to provide mid-range theoretical arguments which may be used to structure existing data and which can be undercut or reinforced by new data. RESULTS AND DISCUSSION: Many barriers and facilitators to integrating care are those of any large-scale organisational change. These include issues relating to leadership, organisational culture, information technology, physician involvement, and availability of resources. However, activities which appear particularly important for delivering integrated care include personal relationships between leaders in different organisations, the scale of planned activities, governance and finance arrangements, support for staff in new roles, and organisational and staff stability. We illustrate our analyses with a 'routemap' which identifies questions that providers may wish to consider when planning interventions to improve the integration of care.

Case management for at-risk elderly patients in the English integrated care pilots: observational study of staff and patient experience and secondary care utilisation.

INTRODUCTION: In 2009, the English Department of Health appointed 16 integrated care pilots which aimed to provide better integrated care. We report the quantitative results from a multi-method evaluation of six of the demonstration projects which used risk profiling tools to identify older people at risk of emergency hospital admission, combined with intensive case management for people identified as at risk. The interventions focused mainly on delivery system redesign and improved clinical information systems, two key elements of Wagner's Chronic Care Model. METHODS: Questionnaires to staff and patients. Difference-in-differences analysis of secondary care utilisation using data on 3646 patients and 17,311 matched controls, and changes in overall secondary care utilisation. RESULTS: Most staff thought that care for their patients had improved. More patients reported having a care plan but they found it significantly harder to see a doctor or nurse of their choice and felt less involved in decisions about their care. Case management interventions were associated with a 9% increase in emergency admissions. We found some evidence of imbalance between cases and controls which could have biased this estimate, but simulations of the possible effect of unobserved confounders showed that it was very unlikely that the sites achieved their goal of reducing emergency admissions. However, we found significant reductions of 21% and 22% in elective admissions and outpatient attendance in the six months following an intervention, and overall inpatient and outpatient costs were significantly reduced by 9% during this period. Area level analyses of whole practice populations suggested that overall outpatient attendances were significantly reduced by 5% two years after the start of the case management schemes. CONCLUSION: Case management may result in improvements in some aspects of care and has the potential to reduce secondary care costs. However, to improve patient experience, case management approaches need to be introduced in a way which respects patients' wishes, for example the ability to see a familiar doctor or nurse.

A novel performance monitoring framework for health research systems: experiences of the National Institute for Health Research in England.

BACKGROUND: The National Institute for Health Research (NIHR) was established in 2006 with the aim of creating an applied health research system embedded within the English National Health Service (NHS). NIHR sought to implement an approach for monitoring its performance that effectively linked early indicators of performance with longer-term research impacts. We attempted to develop and apply a conceptual framework for defining appropriate key performance indicators for NIHR. METHOD: Following a review of relevant literature, a conceptual framework for defining performance indicators for NIHR was developed, based on a hybridisation of the logic model and balanced scorecard approaches. This framework was validated through interviews with key NIHR stakeholders and a pilot in one division of NIHR, before being refined and applied more widely. Indicators were then selected and aggregated to create a basket of indicators aligned to NIHR's strategic goals, which could be reported to NIHR's leadership team on a quarterly basis via an oversight dashboard. RESULTS: Senior health research system managers and practitioners endorsed the conceptual framework developed and reported satisfaction with the breadth and balance of indicators selected for reporting. CONCLUSIONS: The use of the hybrid conceptual framework provides a pragmatic approach to defining performance indicators that are aligned to the strategic aims of a health research system. The particular strength of this framework is its capacity to provide an empirical link, over time, between upstream activities of a health research system and its long-term strategic objectives.

Assessing the Impacts of Revising the Tobacco Products Directive: Study to Support a DG SANCO Impact Assessment.

Tobacco use is one of the largest avoidable causes of morbidity and premature death in the EU. Whilst smoking prevalence in the EU has been declining over the past 30 years, smoking has remained more prevalent among men than women in the EU-27, with some of the new Member States reporting the widest gaps between male and female smokers. For young smokers (13 to 15 years old) this situation is somewhat reversed, with slightly more girls than boys smoking. Against this background, the European Commission Directorate-General for Health and Consumer Protection (DG SANCO) considered a revision of the Tobacco Products Directive 2001/37/EC across five general areas: scope of the directive, labelling requirements, registration and market control fees, ingredients, and sales arrangements. More specifically, the types of policy options under consideration included (but were not limited to): an increase of warning label sizes on the back of packaging to 100%, a restriction for the display of products at retail outlets and an introduction of additional measurement method for TNCO (the modified ISO method) with maximum limits set accordingly. DG SANCO commissioned RAND Europe to provide support in assessing the potential health, macroeconomic, and compliance cost and administrative burden impacts of revising the Tobacco Products Directive. In addition to assessing impacts, the study provides an up-to-date overview of the evidence and basis for current tobacco product regulation that may be of interest to a wider audience interested in tobacco control policies.

A Prize Worth Paying?: Non-Standard Ways to Support and Reward Excellence in Health Research and Development in the UK NHS.

This article describes a study scoping the issues involved in considering the merits of using prizes to support the objectives of the Department of Health Research and Development Directorate (DH R&D). The paper concludes that there is indeed merit in developing incentives to support excellence in health research in addition to "standard" performance management and routine inspection. These could act either to reinforce the signals created by standard metrics (for example, awards recognising the best performers as measured by standard metrics) or they could "fill the gaps" to encourage behaviour not influenced by conventional incentives. This would create an ecosystem to more effectively link reward with motivation, which could deliver benefits for patients and the health care system more widely. Prizes, it is argued, should play a more significant role in the UK health R&D system than in the past, but it is not suggested that they replace existing systems to support high-quality research and development.

Evaluation of DG SANCO Data Management Practices.

The European Commission Health and Consumer Protection Directorate-General (DG SANCO) commissioned RAND Europe to provide support in developing a comprehensive data strategy for DG SANCO that meets the needs of increasingly evidence-based policymaking in the future. This work builds on previous work by RAND Europe conducted for DG SANCO, mapping out international good practice of data management. The work described in this study had two aims: to assess the current data management practices within DG SANCO that relate to the four specific issues identified by DG SANCO: data needs, DG SANCO data sources, key partnerships on data, and data quality; and to develop, on the basis of this review, recommendations for improving DG SANCO's current data management and the definition of DG SANCO's Good Practice Model for Data Strategy. This article presents the findings of RAND Europe's analysis.

Evaluation of UK Integrated Care Pilots: research protocol.

BACKGROUND: In response to concerns that the needs of the aging population for well-integrated care were increasing, the English National Health Service (NHS) appointed 16 Integrated Care Pilots following a national competition. The pilots have a range of aims including development of new organisational structures to support integration, changes in staff roles, reducing unscheduled emergency hospital admissions, reduced length of hospital stay, increasing patient satisfaction, and reducing cost. This paper describes the evaluation of the initiative which has been commissioned. STUDY DESIGN AND DATA COLLECTION METHODS: A mixed methods approach has been adopted including interviews with staff and patients, non-participant observation of meetings, structured written feedback from sites, questionnaires to patients and staff, and analysis of routinely collected hospital utilisation data for patients/service users. The qualitative analysis aims to identify the approaches taken to integration by the sites, the benefits which result, the context in which benefits have resulted, and the mechanisms by which they occur. METHODS OF ANALYSIS: The quantitative analysis adopts a 'difference in differences' approach comparing health care utilisation before and after the intervention with risk-matched controls. The qualitative data analysis adopts a 'theory of change' approach in which we triangulate data from the quantitative analysis with qualitative data in order to describe causal effects (what happens when an independent variable changes) and causal mechanisms (what connects causes to their effects). An economic analysis will identify what incremental resources are required to make integration succeed and how they can be combined efficiently to produce better outcomes for patients. CONCLUSION: This evaluation will produce a portfolio of evidence aimed at strengthening the evidence base for integrated care, and in particular identifying the context in which interventions are likely to be effective. These data will support a series of evaluation judgements aimed at reducing uncertainties about the role of integrated care in improving the efficient and effective delivery of healthcare.