Pragmatic trials and implementation science: grounds for divorce?

BACKGROUND: The paper opens with a brief history of two of the major intellectual components of the recent utilitarian turn in clinical research, namely 'pragmatic trials' and 'implementation science'. The two schools of thought developed independently and the paper scrutinises their mutual compatibilities and incompatibilities, asking: i) what do the leading advocates of pragmatic trials assume about the transfer of research findings to real-world practice and ii) what role pragmatic trials can and should play in the evaluation of implementation science strategies. METHODS: The paper utilises 'explication de texte': i) providing a close reading of the inferential logics contained in major published expositions of the two paradigms, and ii) interrogating the conclusions of a pragmatic trial of an intervention providing guidelines on retinal screening aimed at family practitioners. RESULTS: The paper is in two parts. Part 1 unearths some significant incommensurability - the pragmatic trial literature retains an antiquated view of knowledge transfer and is overly optimistic about the wide applicability the findings of pragmatic trials to 'real world' conditions. Part 2 of the paper outlines an empirical strategy to better penetrate the mechanisms of knowledge transfer and to tackle the issue of the generalisabilty of research findings in implementation science. CONCLUSIONS: Pragmatism, classically, is about problem solving and the melding of perspectives. The core research requirement in implementation science is a fundamental shift from the narrow shoulders of pragmatic trials to a model of explanation building based upon a multi-case, multi-method body of evidence.

Evaluating deceased organ donation: a programme theory approach.

PURPOSE: Organ donation and transplantation services represent a microcosm of modern healthcare organisations. They are complex adaptive systems. They face perpetual problems of matching supply and demand. They operate under fierce time and resource constraints. And yet they have received relatively little attention from a systems perspective. The purpose of this paper is to consider some of the fundamental issues in evaluating, improving and policy reform in such complex systems. DESIGN/METHODOLOGY/APPROACH: The paper advocates an approach based on programme theory evaluation. FINDINGS: The paper explains how the death to donation to transplantation process depends on the accumulation of series of embedded, institutional sub-processes. Evaluators need to be concerned with this whole system rather than with its discrete parts or sectors. Policy makers may expect disappointment if they seek to improve donation rates by applying nudges or administrative reforms at a single point in the implementation chain. ORIGINALITY/VALUE: These services represent concentrated, perfect storms of complexity and the paper offers guidance to practitioners with bio-medical backgrounds on how such services might be evaluated and improved. For the methodological audience the paper caters for the burgeoning interest in programme theory evaluation while illustrating the design phase of this research strategy.

RAMESES publication standards: meta-narrative reviews.

BACKGROUND: Meta-narrative review is one of an emerging menu of new approaches to qualitative and mixed-method systematic review. A meta-narrative review seeks to illuminate a heterogeneous topic area by highlighting the contrasting and complementary ways in which researchers have studied the same or a similar topic. No previous publication standards exist for the reporting of meta-narrative reviews. This publication standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for meta-narrative reviews. METHODS: We (a) collated and summarized existing literature on the principles of good practice in meta-narrative reviews; (b) considered the extent to which these principles had been followed by published reviews, thereby identifying how rigor may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, meta-narrative reviews, policy and/or publishing to produce and iteratively refine a draft set of methodological steps and publication standards; (d) provided real-time support to ongoing meta-narrative reviews and the open-access RAMESES online discussion list so as to capture problems and questions as they arose; and (e) synthesized expert input, evidence review and real-time problem analysis into a definitive set of standards. RESULTS: We identified nine published meta-narrative reviews, provided real-time support to four ongoing reviews and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature, and common questions and challenges into briefing materials for the Delphi panel, comprising 33 members. Within three rounds this panel had reached consensus on 20 key publication standards, with an overall response rate of 90%. CONCLUSION: This project used multiple sources to draw together evidence and expertise in meta-narrative reviews. For each item we have included an explanation for why it is important and guidance on how it might be reported. Meta-narrative review is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further theoretical and methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of meta-narrative reviews.

RAMESES publication standards: realist syntheses.

BACKGROUND: There is growing interest in realist synthesis as an alternative systematic review method. This approach offers the potential to expand the knowledge base in policy-relevant areas - for example, by explaining the success, failure or mixed fortunes of complex interventions. No previous publication standards exist for reporting realist syntheses. This standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for realist systematic reviews. METHODS: We (a) collated and summarized existing literature on the principles of good practice in realist syntheses; (b) considered the extent to which these principles had been followed by published syntheses, thereby identifying how rigor may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, realist research, policy and/or publishing to produce and iteratively refine a draft set of methodological steps and publication standards; (d) provided real-time support to ongoing realist syntheses and the open-access RAMESES online discussion list so as to capture problems and questions as they arose; and (e) synthesized expert input, evidence syntheses and real-time problem analysis into a definitive set of standards. RESULTS: We identified 35 published realist syntheses, provided real-time support to 9 on-going syntheses and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature and common questions and challenges into briefing materials for the Delphi panel, comprising 37 members. Within three rounds this panel had reached consensus on 19 key publication standards, with an overall response rate of 91%. CONCLUSION: This project used multiple sources to develop and draw together evidence and expertise in realist synthesis. For each item we have included an explanation for why it is important and guidance on how it might be reported. Realist synthesis is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of realist syntheses.

RAMESES publication standards: meta-narrative reviews.

BACKGROUND: Meta-narrative review is one of an emerging menu of new approaches to qualitative and mixed-method systematic review. A meta-narrative review seeks to illuminate a heterogeneous topic area by highlighting the contrasting and complementary ways researchers have studied the same or a similar topic. No previous publication standards exist for the reporting of meta-narrative reviews. This publication standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for meta-narrative reviews. DESIGN: A mixed method study synthesising data between 2011 to 2012 from a literature review, online Delphi panel and feedback from training, workshops and email list. METHODS: We: (a) collated and summarized existing literature on the principles of good practice in meta-narrative reviews; (b) considered the extent to which these principles had been followed by published reviews, thereby identifying how rigor may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, meta-narrative reviews, policy, and/or publishing to produce and iteratively refine a draft set of methodological steps, and publication standards; (d) provided real-time support to ongoing meta-narrative reviews and the open-access RAMESES online discussion list so as to capture problems and questions as they arose; and (e) synthesized expert input, evidence review, and real-time problem analysis into a definitive set of standards. RESULTS: We identified nine published meta-narrative reviews, provided real-time support to four ongoing reviews, and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature, and common questions and challenges into briefing materials for the Delphi panel, comprising 33 members. Within three rounds this panel had reached consensus on 20 key publication standards, with an overall response rate of 90%. CONCLUSIONS: This project used multiple sources to draw together evidence and expertise in meta-narrative reviews. For each item we have included an explanation for why it is important and guidance on how it might be reported. Meta-narrative review is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further theoretical and methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of meta-narrative reviews.

RAMESES publication standards: realist syntheses.

BACKGROUND: There is growing interest in realist synthesis as an alternative systematic review method. This approach offers the potential to expand the knowledge base in policy-relevant areas - for example, by explaining the success, failure or mixed fortunes of complex interventions. No previous publication standards exist for reporting realist syntheses. This standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for realist systematic reviews. DESIGN: A mixed method study synthesising data between 2011-2012 from a literature review, online Delphi panel and feedback from training, workshops and email list. METHODS: We: (a) collated and summarized existing literature on the principles of good practice in realist syntheses; (b) considered the extent to which these principles had been followed by published syntheses, thereby identifying how rigour may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, realist research, policy and/or publishing to produce and iteratively refine a draft set of methodological steps and publication standards; (d) provided real-time support to ongoing realist syntheses and the open-access RAMESES online discussion list to capture problems and questions as they arose; and (e) synthesized expert input, evidence syntheses and real-time problem analysis into a definitive set of standards. RESULTS: We identified 35 published realist syntheses, provided real-time support to 9 ongoing syntheses and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature and common questions and challenges into briefing materials for the Delphi panel, comprising 37 members. Within 3 rounds this panel had reached consensus on 19 key publication standards, with an overall response rate of 91%. CONCLUSIONS: This project used multiple sources to develop and draw together evidence and expertise in realist synthesis. For each item we have included an explanation for why it is important and guidance on how it might be reported. Realist synthesis is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of realist syntheses.

Realist methods in medical education research: what are they and what can they contribute?

CONTEXT: Education is a complex intervention which produces different outcomes in different circumstances. Education researchers have long recognised the need to supplement experimental studies of efficacy with a broader range of study designs that will help to unpack the 'how' and 'why' questions and illuminate the many, varied and interdependent mechanisms by which interventions may work (or fail to work) in different contexts. METHODS: One promising approach is realist evaluation, which seeks to establish what works, for whom, in what circumstances, in what respects, to what extent, and why. This paper introduces the realist approach and explains why it is particularly suited to education research. It gives a brief introduction to the philosophical assumptions underlying realist methods and outlines key principles of realist evaluation (designed for empirical studies) and realist review (the application of realist methods to secondary research). DISCUSSION: The paper warns that realist approaches are not a panacea and lists the circumstances in which they are likely to be particularly useful.

Non-pharmaceutical interventions during COVID-19 in the UK and Spain: a rapid realist review.

The paper is located at the crossroads of two modern intellectual movements. The first, evidence-based policy, seeks to locate vital information that will inform and improve key policy decisions on such matters as population health, social welfare, and human wellbeing. The second, complexity theory, describes the nature of the social world and perceives human action as persistently adaptive and social institutions as incessantly self-transformative. The first assumes that policies and programmes can achieve sufficient control to meet specific and measurable objectives. The second assumes that social actions are sufficiently capricious so that the society never conforms to anyone's plans - even those of the most powerful. The unparalleled resources committed to control the unprecedented attack of the COVID-19 pandemic are the epitome of complexity. The long struggle to contain the virus thus constitutes an ideal test bed to investigate this paradigmatic split. The paper undertakes this mission - focusing specifically on the effectiveness non-pharmaceutical interventions and examining evidence from the UK and Spain.

Protocol--realist and meta-narrative evidence synthesis: evolving standards (RAMESES).

BACKGROUND: There is growing interest in theory-driven, qualitative and mixed-method approaches to systematic review as an alternative to (or to extend and supplement) conventional Cochrane-style reviews. These approaches offer the potential to expand the knowledge base in policy-relevant areas - for example by explaining the success, failure or mixed fortunes of complex interventions. However, the quality of such reviews can be difficult to assess. This study aims to produce methodological guidance, publication standards and training resources for those seeking to use the realist and/or meta-narrative approach to systematic review. METHODS/DESIGN: We will: [a] collate and summarise existing literature on the principles of good practice in realist and meta-narrative systematic review; [b] consider the extent to which these principles have been followed by published and in-progress reviews, thereby identifying how rigour may be lost and how existing methods could be improved; [c] using an online Delphi method with an interdisciplinary panel of experts from academia and policy, produce a draft set of methodological steps and publication standards; [d] produce training materials with learning outcomes linked to these steps; [e] pilot these standards and training materials prospectively on real reviews-in-progress, capturing methodological and other challenges as they arise; [f] synthesise expert input, evidence review and real-time problem analysis into more definitive guidance and standards; [g] disseminate outputs to audiences in academia and policy. The outputs of the study will be threefold:1. Quality standards and methodological guidance for realist and meta-narrative reviews for use by researchers, research sponsors, students and supervisors2. A 'RAMESES' (Realist and Meta-review Evidence Synthesis: Evolving Standards) statement (comparable to CONSORT or PRISMA) of publication standards for such reviews, published in an open-access academic journal.3. A training module for researchers, including learning outcomes, outline course materials and assessment criteria. DISCUSSION: Realist and meta-narrative review are relatively new approaches to systematic review whose overall place in the secondary research toolkit is not yet fully established. As with all secondary research methods, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency of studies.

Policy guidance on threats to legislative interventions in public health: a realist synthesis.

BACKGROUND: Legislation is one of the most powerful weapons for improving population health and is often used by policy and decision makers. Little research exists to guide them as to whether legislation is feasible and/or will succeed. We aimed to produce a coherent and transferable evidence based framework of threats to legislative interventions to assist the decision making process and to test this through the 'case study' of legislation to ban smoking in cars carrying children. METHODS: We conceptualised legislative interventions as a complex social interventions and so used the realist synthesis method to systematically review the literature for evidence. 99 articles were found through searches on five electronic databases (MEDLINE, HMIC, EMBASE, PsychINFO, Social Policy and Practice) and iterative purposive searching. Our initial searches sought any studies that contained information on smoking in vehicles carrying children. Throughout the review we continued where needed to search for additional studies of any type that would conceptually contribute to helping build and/or test our framework. RESULTS: Our framework identified a series of transferable threats to public health legislation. When applied to smoking bans in vehicles; problem misidentification; public support; opposition; and enforcement issues were particularly prominent threats. Our framework enabled us to understand and explain the nature of each threat and to infer the most likely outcome if such legislation were to be proposed in a jurisdiction where no such ban existed. Specifically, the micro-environment of a vehicle can contain highly hazardous levels of second hand smoke. Public support for such legislation is high amongst smokers and non-smokers and their underlying motivations were very similar - wanting to practice the Millian principle of protecting children from harm. Evidence indicated that the tobacco industry was not likely to oppose legislation and arguments that such a law would be 'unenforceable' were unfounded. CONCLUSION: It is possible to develop a coherent and transferable evidence based framework of the ideas and assumptions behind the threats to legislative intervention that may assist policy and decision makers to analyse and judge if legislation is feasible and/or likely to succeed.

A new workforce in the making? A case study of strategic human resource management in a whole-system change effort in healthcare.

PURPOSE: This paper seeks to describe the exploration of human resource issues in one large-scale program of innovation in healthcare. It is informed by established theories of management in the workplace and a multi-level model of diffusion of innovations. DESIGN/METHODOLOGY/APPROACH: A realist approach was used based on interviews, ethnographic observation and documentary analysis. FINDINGS: Five main approaches ("theories of change") were adopted to develop and support the workforce: recruiting staff with skills in service transformation; redesigning roles and creating new roles; enhancing workforce planning; linking staff development to service needs; creating opportunities for shared learning and knowledge exchange. Each had differing levels of success. PRACTICAL IMPLICATIONS: The paper includes HR implications for the modernisation of a complex service organisation. ORIGINALITY/VALUE: This is the first time a realist evaluation of a complex health modernisation initiative has been undertaken.

Internet-based medical education: a realist review of what works, for whom and in what circumstances.

BACKGROUND: Educational courses for doctors and medical students are increasingly offered via the Internet. Despite much research, course developers remain unsure about what (if anything) to offer online and how. Prospective learners lack evidence-based guidance on how to choose between the options on offer. We aimed to produce theory driven criteria to guide the development and evaluation of Internet-based medical courses. METHODS: Realist review - a qualitative systematic review method whose goal is to identify and explain the interaction between context, mechanism and outcome. We searched 15 electronic databases and references of included articles, seeking to identify theoretical models of how the Internet might support learning from empirical studies which (a) used the Internet to support learning, (b) involved doctors or medical students; and (c) reported a formal evaluation. All study designs and outcomes were considered. Using immersion and interpretation, we tested theories by considering how well they explained the different outcomes achieved in different educational contexts. RESULTS: 249 papers met our inclusion criteria. We identified two main theories of the course-in-context that explained variation in learners' satisfaction and outcomes: Davis's Technology Acceptance Model and Laurillard's model of interactive dialogue. Learners were more likely to accept a course if it offered a perceived advantage over available non-Internet alternatives, was easy to use technically, and compatible with their values and norms. 'Interactivity' led to effective learning only if learners were able to enter into a dialogue - with a tutor, fellow students or virtual tutorials - and gain formative feedback. CONCLUSIONS: Different modes of course delivery suit different learners in different contexts. When designing or choosing an Internet-based course, attention must be given to the fit between its technical attributes and learners' needs and priorities; and to ways of providing meaningful interaction. We offer a preliminary set of questions to aid course developers and learners consider these issues.

How do you modernize a health service? A realist evaluation of whole-scale transformation in london.

CONTEXT: Large-scale, whole-systems interventions in health care require imaginative approaches to evaluation that go beyond assessing progress against predefined goals and milestones. This project evaluated a major change effort in inner London, funded by a charitable donation of approximately $21 million, which spanned four large health care organizations, covered three services (stroke, kidney, and sexual health), and sought to "modernize" these services with a view to making health care more efficient, effective, and patient centered. METHODS: This organizational case study draws on the principles of realist evaluation, a largely qualitative approach that is centrally concerned with testing and refining program theories by exploring the complex and dynamic interaction among context, mechanism, and outcome. This approach used multiple data sources and methods in a pragmatic and reflexive manner to build a picture of the case and follow its fortunes over the three-year study period. The methods included ethnographic observation, semistructured interviews, and scrutiny of documents and other contemporaneous materials. As well as providing ongoing formative feedback to the change teams in specific areas of activity, we undertook a more abstract, interpretive analysis, which explored the context-mechanism-outcome relationship using the guiding question "what works, for whom, under what circumstances?" FINDINGS: In this example of large-scale service transformation, numerous projects and subprojects emerged, fed into one another, and evolved over time. Six broad mechanisms appeared to be driving the efforts of change agents: integrating services across providers, finding and using evidence, involving service users in the modernization effort, supporting self-care, developing the workforce, and extending the range of services. Within each of these mechanisms, different teams chose widely differing approaches and met with differing success. The realist analysis of the fortunes of different subprojects identified aspects of context and mechanism that accounted for observed outcomes (both intended and unintended). CONCLUSIONS: This study was one of the first applications of realist evaluation to a large-scale change effort in health care. Even when an ambitious change program shifts from its original goals and meets unforeseen challenges (indeed, precisely because the program morphs and adapts over time), realist evaluation can draw useful lessons about how particular preconditions make particular outcomes more likely, even though it cannot produce predictive guidance or a simple recipe for success. Noting recent calls by others for the greater use of realist evaluation in health care, this article considers some of the challenges and limitations of this method in the light of this experience and suggests that its use will require some fundamental changes in the worldview of some health services researchers.

The "pragmatic trial": An essentially contested concept?

For over 50 years, clinical research methodology has wrestled with the problem of the lack of correspondence between tests of treatments and applications of treatments. The former comprise of trials featuring scrupulous control of patient eligibility, treatment compliance, clinician expertise, follow-up intensity, and so on. In applying a validated treatment, the practitioner has to confront considerable real-world variation in potential patients and in implementation regimes. The remedy, going by the name of "pragmatic trials," is to conduct clinical trials in conditions corresponding more closely to everyday practice. This solution has proved easier to utter than to execute, and the paper reviews the extensive literature on pragmatic trials, seeking to assess whether it has terminated in clarity or contestation.

Guidance on guidelines: Understanding the evidence on the uptake of health care guidelines.

RATIONALE: Regardless of health issue, health sector, patient condition, or treatment modality, the chances are that provision is supported by "a guideline" making professionally endorsed recommendations on best practice. Against this background, research has proliferated seeking to evaluate how effectively such guidance is followed. These investigations paint a gloomy picture with many a guideline prompting lip service, inattention, and even opposition. This predicament has prompted a further literature on how to improve the uptake of guidelines, and this paper considers how to draw together lessons from these inquiries. METHODS: This huge body of material presents a considerable challenge for research synthesis, and this paper produces a critical, methodological comparison of 2 types of review attempting to meet that task. Firstly, it provides an overview of the current orthodoxy, namely, "thematic reviews," which aggregate and enumerate the "barriers and facilitators" to guideline implementation. It then outlines a "realist synthesis," focussing on testing the "programme theories" that practitioners have devised to improve guideline uptake. RESULTS: Thematic reviews aim to provide a definitive, comprehensive catalogue of the facilitators and barriers to guideline implementation. As such, they present a restatement of the underlying problems rather than an improvement strategy. The realist approach assumes that the incorporation of any guideline into current practice will produce unintended system strains as different stakeholders wrestle over responsibilities. These distortions will prompt supplementary revisions to guidelines, which in turn beget further strains. Realist reviews follow this dynamic understanding of organisational change. CONCLUSIONS: Health care decision makers operate in systems that are awash with guidelines. But guidelines only have paper authority. Managers do not need a checklist of their pros and cons, because the fate of guidelines depends on their reception rather than their production. They do need decision support on how to engineer and reengineer guidelines so they dovetail with evolving systems of health care delivery.

Realist review--a new method of systematic review designed for complex policy interventions.

Evidence-based policy is a dominant theme in contemporary public services but the practical realities and challenges involved in using evidence in policy-making are formidable. Part of the problem is one of complexity. In health services and other public services, we are dealing with complex social interventions which act on complex social systems--things like league tables, performance measures, regulation and inspection, or funding reforms. These are not 'magic bullets' which will always hit their target, but programmes whose effects are crucially dependent on context and implementation. Traditional methods of review focus on measuring and reporting on programme effectiveness, often find that the evidence is mixed or conflicting, and provide little or no clue as to why the intervention worked or did not work when applied in different contexts or circumstances, deployed by different stakeholders, or used for different purposes. This paper offers a model of research synthesis which is designed to work with complex social interventions or programmes, and which is based on the emerging 'realist' approach to evaluation. It provides an explanatory analysis aimed at discerning what works for whom, in what circumstances, in what respects and how. The first step is to make explicit the programme theory (or theories)--the underlying assumptions about how an intervention is meant to work and what impacts it is expected to have. We then look for empirical evidence to populate this theoretical framework, supporting, contradicting or modifying the programme theories as it goes. The results of the review combine theoretical understanding and empirical evidence, and focus on explaining the relationship between the context in which the intervention is applied, the mechanisms by which it works and the outcomes which are produced. The aim is to enable decision-makers to reach a deeper understanding of the intervention and how it can be made to work most effectively. Realist review does not provide simple answers to complex questions. It will not tell policy-makers or managers whether something works or not, but will provide the policy and practice community with the kind of rich, detailed and highly practical understanding of complex social interventions which is likely to be of much more use to them when planning and implementing programmes at a national, regional or local level.

Protocol--the RAMESES II study: developing guidance and reporting standards for realist evaluation.

INTRODUCTION: Realist evaluation is an increasingly popular methodology in health services research. For realist evaluations (RE) this project aims to: develop quality and reporting standards and training materials; build capacity for undertaking and critically evaluating them; produce resources and training materials for lay participants, and those seeking to involve them. METHODS: To achieve our aims, we will: (1) Establish management and governance infrastructure; (2) Recruit an interdisciplinary Delphi panel of 35 participants with diverse relevant experience of RE; (3) Summarise current literature and expert opinion on best practice in RE; (4) Run an online Delphi panel to generate and refine items for quality and reporting standards; (5) Capture 'real world' experiences and challenges of RE-for example, by providing ongoing support to realist evaluations, hosting the RAMESES JISCmail list on realist research, and feeding problems and insights from these into the deliberations of the Delphi panel; (6) Produce quality and reporting standards; (7) Collate examples of the learning and training needs of researchers, students, reviewers and lay members in relation to RE; (8) Develop, deliver and evaluate training materials for RE and deliver training workshops; and (9) Develop and evaluate information and resources for patients and other lay participants in RE (eg, draft template information sheets and model consent forms) and; (10) Disseminate training materials and other resources.Planned outputs: (1) Quality and reporting standards and training materials for RE. (2) Methodological support for RE. (3) Increase in capacity to support and evaluate RE. (4) Accessible, plain-English resources for patients and the public participating in RE. DISCUSSION: The realist evaluation is a relatively new approach to evaluation and its overall place in the is not yet fully established. As with all primary research approaches, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency.