A systematic review of studies reporting the development of core outcome sets for use in routine care.

OBJECTIVES: Core outcome sets (COS) represent the minimum health outcomes to be measured for a given health condition. Interest is growing in using COS within routine care to support delivery of patient-focused care. This review aims to systematically map COS developed for routine care to understand their scope, stakeholder involvement, and development methods. METHODS: Medline (Ovid), Scopus, and Web of Science Core collection were searched for studies reporting development of COS for routine care. Data on scope, methods, and stakeholder groups were analyzed in subgroups defined by setting. RESULTS: Screening 25,301 records identified 262 COS: 164 for routine care only and 98 for routine care and research. Nearly half of the COS (112/254, 44%) were developed with patients, alongside input from experts in registries, insurance, legal, outcomes measurement, and performance management. Research publications were often searched to generate an initial list of outcomes (115/198, 58%) with few searching routine health records (47/198, 24%). CONCLUSION: An increasing number of COS is being developed for routine care. Although involvement of patient stakeholders has increased in recent years, further improvements are needed. Methodology and scope are broadly similar to COS for research but implementation of the final set is a greater consideration during development.

Review finds core outcome set uptake in new studies and systematic reviews needs improvement.

OBJECTIVES: To review evidence about the uptake of core outcome sets (COS). A COS is an agreed standardized set of outcomes that should be measured and reported, as a minimum, in all clinical trials in a specific area of health or healthcare. STUDY DESIGN AND SETTING: This article provides an analysis of what is known about the uptake of COS in research. Similarities between COS and outcomes recommended by stakeholders in the evidence ecosystem is reviewed and actions taken by them to facilitate COS uptake described. RESULTS: COS uptake is low in most research areas. Common facilitators relate to trialist awareness and understanding. Common barriers were not including in the development process all specialties that might use the COS and the lack of recommendations for how to measure the outcomes. Increasingly, COS developers are considering strategies for promoting uptake earlier in the process, including actions beyond traditional dissemination approaches. An overlap between COS and outcomes in regulatory documents and health technology assessments is good. An increasing number and variety of organizations are recommending COS be considered. CONCLUSION: We suggest actions for various stakeholders for improving COS uptake. Research is needed to assess the impact of these actions to identify effective evidence-based strategies.

Inclusion of participants from low-income and middle-income countries in core outcome sets development: a systematic review.

OBJECTIVE: Our study aims to describe differences or similarities in the scope, participant characteristics and methods used in core outcome sets (COS) development when only participants from high-income countries (HICs) were involved compared with when participants from low-income and middle-income countries (LMICs) were also involved. DESIGN: Systematic review. DATA SOURCES: Annual Core Outcome Measures in Effectiveness Trials systematic reviews of COS which are updated based on SCOPUS and MEDLINE, searches. The latest systematic review included studies published up to the end of 2019. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: We included studies reporting development of a COS for use in research regardless of age, health condition or setting. Studies reporting the development of a COS for patient-reported outcomes or adverse events or complications were also included. DATA EXTRACTION AND SYNTHESIS: Data were extracted in relation to scope of the COS study, participant categories and the methods used in outcome selection. RESULTS: Studies describing 370 COS were identified in the database. Of these, 75 (20%) included participants from LMICs. Only four COS were initiated from an LMIC setting. More than half of COS with LMIC participants were developed in the last 5 years. Cancer and rheumatology were the dominant disease domains. Overall, over 259 (70%) of COS explicitly reported including clinical experts; this was higher where LMIC participants were also included 340 (92%). Most LMIC participants were from China, Brazil and South Africa. Mixed methods for consensus building were used across the two settings. CONCLUSION: Progress has been made in including LMIC participants in the development of COS, however, there is a need to explore how to enable initiation of COS development from a range of LMIC settings, how to ensure prioritisation of COS that better reflects the burden of disease in these contexts and how to improve public participation from LMICs.

Choosing important health outcomes for comparative effectiveness research: 6th annual update to a systematic review of core outcome sets for research.

BACKGROUND: An annual update to a systematic review of core outcome sets (COS) for research ensures that the COMET database is up-to-date. The aims of this study were to: (i) identify COS that were published or indexed in 2019 and to describe the methodological approaches used in these studies; (ii) investigate whether children have been included as participants in published COS development studies, and which methods have been used to facilitate their participation; iii) update a previous exercise to identify COS relevant to the most burdensome global diseases and injuries. METHODS: MEDLINE and SCOPUS were searched to identify studies published or indexed between (and inclusive of) January 2019 and December 2019. Automated screening methods were used to rank the citations in order of relevance; the top 25% in ranked priority order were screened for eligibility. COS were assessed against each of the Core Outcome Set-STAndards for Development (COS-STAD). A search of the COMET database was undertaken to identify COS relevant to the 25 leading causes of disease burden. RESULTS: Thirty-three studies, describing the development of 37 COS, were included in this update. These studies have been added to the COMET database, which now contains 370 published (1981-2019) COS studies for clinical research. Six (18%) of the 33 studies in this update were deemed to have met all of the minimum standards for COS development (range = 4 to 12 criteria, median = 9 criteria). Of the 370 COS studies published to date, 82 COS have been developed for paediatric health conditions and children would have been eligible to participate in 68/82 of these studies. Eleven of these 68 (16%) COS studies have included children as participants within the development process, most commonly through participation in Delphi surveys. Relevant COS were identified for 22/25 leading causes of global disease burden. CONCLUSION: There has been a demonstrated increase in COS developed for both research and routine practice, and consistently high inclusion of patient participants. COS developed for paediatric conditions need to further incorporate the perspectives of children, alongside parents and other adults, and adopt research methods fit for this purpose. COS developers should consider the gaps identified in this update as priorities for COS development.

Methods used to assess outcome consistency in clinical studies: A literature-based evaluation.

Evaluation studies of outcomes used in clinical research and their consistency are appearing more frequently in the literature, as a key part of the core outcome set (COS) development. Current guidance suggests such evaluation studies should use systematic review methodology as their default. We aimed to examine the methods used. We searched the Core Outcome Measures in Effectiveness Trials (COMET) database (up to May 2019) supplementing it with additional resources. We included evaluation studies of outcome consistency in clinical studies across health subjects and used a subset of A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 (items 1-9) to assess their methods. Of 93 included evaluation studies of outcome consistency (90 full reports, three summaries), 91% (85/93) reported performing literature searches in at least one bibliographic database, and 79% (73/93) was labelled as a "systematic review". The evaluations varied in terms of satisfying AMSTAR 2 criteria, such that 81/93 (87%) had implemented PICO in the research question, whereas only 5/93 (6%) had included the exclusions list. None of the evaluation studies explained how inconsistency of outcomes was detected, however, 80/90 (88%) concluded inconsistency in individual outcomes (66%, 55/90) or outcome domains (20%, 18/90). Methods used in evaluation studies of outcome consistency in clinical studies differed considerably. Despite frequent being labelled as a "systematic review", adoption of systematic review methodology is selective. While the impact on COS development is unknown, authors of these studies should refrain from labelling them as "systematic review" and focus on ensuring that the methods used to generate the different outcomes and outcome domains are reported transparently.

Choosing important health outcomes for comparative effectiveness research: 5th annual update to a systematic review of core outcome sets for research.

BACKGROUND: A systematic review of core outcome sets (COS) for research is updated annually to populate an online database. It is a resource intensive review to do annually but automation techniques have potential to aid the process. The production of guidance and standards in COS development means that there is now an expectation that COS are being developed and reported to a higher standard. This is the fifth update to the systematic review and will explore these issues. METHODS: Searches were carried out to identify studies published or indexed in 2018. Automated screening methods were used to rank the citations in order of relevance. The cut-off for screening was set to the top 25% in ranked priority order, following development and validation of the algorithm. Studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. COS were assessed against each of the Core Outcome Set-STAndards for Development (COS-STAD). RESULTS: Thirty studies describing the development of 44 COS were included in this update. Six COS (20%) were deemed to have met all 12 criteria representing the 11 minimum standards for COS development (range = 4 to 12 criteria, median = 10 criteria). All 30 COS studies met all four minimum standards for scope. Twenty-one (70%) COS met all three minimum standards for stakeholders. Twenty-three studies (77%) included patients with the condition or their representatives. The number of countries involved in the development of COS ranged from 1 to 39 (median = 10). Six studies (20%) met all four minimum standards [five criteria] for the consensus process. CONCLUSION: Automated ranking was successfully used to assist the screening process and reduce the workload of this systematic review update. With the provision of guidelines, COS are better reported and being developed to a higher standard.

Evaluation of an automatic article selection method for timelier updates of the Comet Core Outcome Set database.

Curated databases of scientific literature play an important role in helping researchers find relevant literature, but populating such databases is a labour intensive and time-consuming process. One such database is the freely accessible Comet Core Outcome Set database, which was originally populated using manual screening in an annually updated systematic review. In order to reduce the workload and facilitate more timely updates we are evaluating machine learning methods to reduce the number of references needed to screen. In this study we have evaluated a machine learning approach based on logistic regression to automatically rank the candidate articles. Data from the original systematic review and its four first review updates were used to train the model and evaluate performance. We estimated that using automatic screening would yield a workload reduction of at least 75% while keeping the number of missed references around 2%. We judged this to be an acceptable trade-off for this systematic review, and the method is now being used for the next round of the Comet database update.

Pain Measurement in Rheumatic and Musculoskeletal Diseases: Where To Go from Here? Report from a Special Interest Group at OMERACT 2018.

OBJECTIVE: Establishing a research agenda on standardizing pain measurement in clinical trials in rheumatic and musculoskeletal diseases (RMD). METHODS: Discussion during a meeting at the Outcome Measures in Rheumatology (OMERACT) 2018, prepared by a systematic review of existing core outcome sets and a patient online survey. RESULTS: Several key questions were debated: Is pain a symptom or a disease? Are pain core (sub)domains consistent across RMD? How to account for pain mechanistic descriptors (e.g., central sensitization) in pain measurement? CONCLUSION: Characterizing and assessing the spectrum of pain experience across RMD in a standardized fashion is the objective of the OMERACT Pain Working Group.

Improvement was needed in the standards of development for cancer core outcome sets.

OBJECTIVE: The Core Outcome Set-STAndards for Development (COS-STAD) contains 11 standards (12 criteria) that are deemed to be the minimum design recommendations for all core outcome set (COS) development projects. Cancer is currently the disease area with the highest number of published COSs and is a major cause of worldwide morbidity and mortality. The aim of this study was to provide a baseline of cancer COS standards. STUDY DESIGN AND SETTING: Systematic reviews of COSs have identified 307 published COS studies. Cancer COSs were eligible for inclusion. Two reviewers independently assessed each of the COSs against the 12 criteria. RESULTS: Forty-nine cancer COSs were included; none met all 12 criteria representing the 11 minimum standards assessed in this study (range = 4-11 criteria, median = 6 criteria). All studies met the four scope standards, eight (16%) met all three standards for stakeholders involved, and two (4%) met all four standards for consensus process standards. CONCLUSION: With the exception of "scope" specification, there is much need for improvement. Poor reporting often made it challenging to assess whether minimum standards were met. The consensus process criteria were most difficult to assess, particularly those that required an assessment of being a priori. This is the first application of COS-STAD criteria to studies that have developed COSs and provides a baseline of cancer COS standards of development.

Higher number of items associated with significantly lower response rates in COS Delphi surveys.

OBJECTIVES: The Delphi method is commonly used to achieve consensus in core outcome set (COS) development. It is important to try to maximize response rates to Delphi studies and minimize attrition rates and potential for bias. The factors that impact response rates in a Delphi study used for COS development are unknown. The objective of this study was to explore the impact of design characteristics on response rates in Delphi surveys within COS development. METHODS: Published and ongoing studies that included Delphi to develop a COS were eligible. Second round voting response rates were analyzed, and multilevel linear regression was conducted to investigate whether design characteristics were associated with the response rate. RESULTS: Thirty-one studies were included. Two characteristics were significantly associated with a lower response rate: larger panels and studies with more items included. CONCLUSION: COS developers should pay attention to methods when designing a COS development study; in particular, the size of the panels and the size of the list of outcomes. We identified other potential design characteristics that might influence response rates but were unable to explore them in this analysis. These should be reported in future reports to allow for further investigation.

Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research.

BACKGROUND: The Core Outcome Measures in Effectiveness Trials (COMET) database is a publically available, searchable repository of published and ongoing core outcome set (COS) studies. An annual systematic review update is carried out to maintain the currency of database content. METHODS: The methods used in the fourth update of the systematic review followed the same approach used in the original review and previous updates. Studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. Searches were carried out in March 2018 to identify studies that had been published or indexed between January 2017 and the end of December 2017. RESULTS: Forty-eight new studies, describing the development of 56 COS, were included. There has been an increase in the number of studies clearly specifying the scope of the COS in terms of the population (n = 43, 90%) and intervention (n = 48, 100%) characteristics. Public participation has continued to rise with over half (n = 27, 56%) of studies in the current review including input from members of the public. The rate of inclusion of all stakeholder groups has increased, in particular participation from non-clinical research experts has risen from 32% (mean average in previous reviews) to 62% (n = 29). Input from participants located in Australasia (n = 17; 41%), Asia (n = 18; 44%), South America (n = 13; 32%) and Africa (n = 7; 17%) have all increased since the previous reviews. CONCLUSION: This update included a pronounced increase in the number of new COS identified compared to the previous three updates. There was an improvement in the reporting of the scope, stakeholder participants and methods used. Furthermore, there has been an increase in participation from Australasia, Asia, South America and Africa. These advancements are reflective of the efforts made in recent years to raise awareness about the need for COS development and uptake, as well as developments in COS methodology.

Choosing important health outcomes for comparative effectiveness research: An updated systematic review and involvement of low and middle income countries.

BACKGROUND: Core outcome sets (COS) comprise a minimum set of outcomes that should be measured and reported in all trials for a specific health condition. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative maintains an up to date, publicly accessible online database of published and ongoing COS. An annual systematic review update is an important part of this process. METHODS: This review employed the same, multifaceted approach that was used in the original review and the previous two updates. This approach has identified studies that sought to determine which outcomes/domains to measure in clinical trials of a specific condition. This update includes an analysis of the inclusion of participants from low and middle income countries (LMICs) as identified by the OECD, in these COS. RESULTS: Eighteen publications, relating to 15 new studies describing the development of 15 COS, were eligible for inclusion in the review. Results show an increase in the use of mixed methods, including Delphi surveys. Clinical experts remain the most common stakeholder group involved. Overall, only 16% of the 259 COS studies published up to the end of 2016 have included participants from LMICs. CONCLUSION: This review highlights opportunities for greater public participation in COS development and the involvement of stakeholders from a wider range of geographical settings, in particular LMICs.

The COMET Handbook: version 1.0.

The selection of appropriate outcomes is crucial when designing clinical trials in order to compare the effects of different interventions directly. For the findings to influence policy and practice, the outcomes need to be relevant and important to key stakeholders including patients and the public, health care professionals and others making decisions about health care. It is now widely acknowledged that insufficient attention has been paid to the choice of outcomes measured in clinical trials. Researchers are increasingly addressing this issue through the development and use of a core outcome set, an agreed standardised collection of outcomes which should be measured and reported, as a minimum, in all trials for a specific clinical area.Accumulating work in this area has identified the need for guidance on the development, implementation, evaluation and updating of core outcome sets. This Handbook, developed by the COMET Initiative, brings together current thinking and methodological research regarding those issues. We recommend a four-step process to develop a core outcome set. The aim is to update the contents of the Handbook as further research is identified.

Improving core outcome set development: qualitative interviews with developers provided pointers to inform guidance.

OBJECTIVES: The objective of the study was to explore core outcome set (COS) developers' experiences of their work to inform methodological guidance on COS development and identify areas for future methodological research. STUDY DESIGN AND SETTING: Semistructured, audio-recorded interviews with a purposive sample of 32 COS developers. Analysis of transcribed interviews was informed by the constant comparative method and framework analysis. RESULTS: Developers found COS development to be challenging, particularly in relation to patient participation and accessing funding. Their accounts raised fundamental questions about the status of COS development and whether it is consultation or research. Developers emphasized how the absence of guidance had affected their work and identified areas where guidance or evidence about COS development would be useful including, patient participation, ethics, international development, and implementation. They particularly wanted guidance on systematic reviews, Delphi, and consensus meetings. CONCLUSION: The findings raise important questions about the funding, status, and process of COS development and indicate ways that it could be strengthened. Guidance could help developers to strengthen their work, but over specification could threaten quality in COS development. Guidance should therefore highlight common issues to consider and encourage tailoring of COS development to the context and circumstances of particular COS.

Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and Identification of Gaps.

BACKGROUND: The COMET (Core Outcome Measures in Effectiveness Trials) Initiative promotes the development and application of core outcome sets (COS), including relevant studies in an online database. In order to keep the database current, an annual search of the literature is undertaken. This study aimed to update a previous systematic review, in order to identify any further studies where a COS has been developed. Furthermore, no prioritization for COS development has previously been undertaken, therefore this study also aimed to identify COS relevant to the world's most prevalent health conditions. METHODS: The methods used in this updated review followed the same approach used in the original review and the previous update. A survey was also sent to the corresponding authors of COS identified for inclusion in this review, to ascertain what lessons they had learnt from developing their COS. Additionally, the COMET database was searched to identify COS that might be relevant to the conditions with the highest global prevalence. RESULTS: Twenty-five reports relating to 22 new studies were eligible for inclusion in the review. Further improvements were identified in relation to the description of the scope of the COS, use of the Delphi technique, and the inclusion of patient participants within the development process. Additionally, 33 published and ongoing COS were identified for 13 of the world's most prevalent conditions. CONCLUSION: The development of a reporting guideline and minimum standards should contribute towards future improvements in development and reporting of COS. This study has also described a first approach to identifying gaps in existing COS, and to priority setting in this area. Important gaps have been identified, on the basis of global burden of disease, and the development and application of COS in these areas should be considered a priority.

Core Outcome Set-STAndards for Reporting: The COS-STAR Statement.

BACKGROUND: Core outcome sets (COS) can enhance the relevance of research by ensuring that outcomes of importance to health service users and other people making choices about health care in a particular topic area are measured routinely. Over 200 COS to date have been developed, but the clarity of these reports is suboptimal. COS studies will not achieve their goal if reports of COS are not complete and transparent. METHODS AND FINDINGS: In recognition of these issues, an international group that included experienced COS developers, methodologists, journal editors, potential users of COS (clinical trialists, systematic reviewers, and clinical guideline developers), and patient representatives developed the Core Outcome Set-STAndards for Reporting (COS-STAR) Statement as a reporting guideline for COS studies. The developmental process consisted of an initial reporting item generation stage and a two-round Delphi survey involving nearly 200 participants representing key stakeholder groups, followed by a consensus meeting. The COS-STAR Statement consists of a checklist of 18 items considered essential for transparent and complete reporting in all COS studies. The checklist items focus on the introduction, methods, results, and discussion section of a manuscript describing the development of a particular COS. A limitation of the COS-STAR Statement is that it was developed without representative views of low- and middle-income countries. COS have equal relevance to studies conducted in these areas, and, subsequently, this guideline may need to evolve over time to encompass any additional challenges from developing COS in these areas. CONCLUSIONS: With many ongoing COS studies underway, the COS-STAR Statement should be a helpful resource to improve the reporting of COS studies for the benefit of all COS users.

Improving the relevance and consistency of outcomes in comparative effectiveness research.

Policy makers have clearly indicated--through heavy investment in the Patient Centered Outcomes Research Institute--that reporting outcomes that are meaningful to patients is crucial for improvement in healthcare delivery and cost reduction. Better interpretation and generalizability of clinical research results that incorporate patient-centered outcomes research can be achieved by accelerating the development and uptake of core outcome sets (COS). COS provide a standardized minimum set of the outcomes that should be measured and reported in all clinical trials of a specific condition. The level of activity around COS has increased significantly over the past decade, with substantial progress in several clinical domains. However, there are many important clinical conditions for which high-quality COS have not been developed and there are limited resources and capacity with which to develop them. We believe that meaningful progress toward the goals behind the significant investments in patient-centered outcomes research and comparative effectiveness research will depend on a serious effort to address these issues.