IDEAL-D Framework for Device Innovation: A Consensus Statement on the Preclinical Stage.

OBJECTIVE: To extend the IDEAL framework for device innovation, IDEAL-D, to include the preclinical stage of development (stage 0). BACKGROUND: In previous work, the IDEAL collaboration has proposed frameworks for new surgical techniques and complex therapeutic technologies, the central tenet being that development and evaluation can and should proceed together in an ordered and logical manner that balances innovation and safety. METHODS: Following agreement at the IDEAL Collaboration Council, a multidisciplinary working group was formed comprising 12 representatives from healthcare, academia, industry, and a patient advocate. The group conducted a series of discussions following the principles used in the development of the original IDEAL framework. Importantly, IDEAL aims for maximal transparency, optimal validity in the evaluation of primary effects, and minimization of potential risk to patients or others. The proposals were subjected to further review and editing by members of the IDEAL Council before a final consensus version was adopted. RESULTS: In considering which studies are required before a first-in-human study, we have: (1) classified devices according to what they do and the risks they carry, (2) classified studies according to what they show about the device, and (3) made recommendations based on the principle that the more invasive and high risk a device is, the greater proof required of their safety and effectiveness before progression to clinical studies (stage 1). CONCLUSIONS: The proposed recommendations for preclinical evaluation of medical devices represent a proportionate and pragmatic approach that balances the de-risking of first-in-human translational studies against the benefits of rapid translation of new devices into clinical practice.

The IDEAL Reporting Guidelines: A Delphi Consensus Statement Stage Specific Recommendations for Reporting the Evaluation of Surgical Innovation.

OBJECTIVE: The aim of this study was to define reporting standards for IDEAL format studies. BACKGROUND: The IDEAL Framework and Recommendations establish an integrated pathway for evaluation of new surgical techniques and complex therapeutic technologies. However guidance on implementation has been incomplete, and incorrect use is commonly seen. We describe the consensus development of reporting guidelines for the IDEAL stages, and plans for their dissemination and evaluation. METHODS: Using the EQUATOR Network recommendations, participants with knowledge of IDEAL were surveyed to determine which IDEAL stages needed reporting guidelines. Draft checklists for stages 1, 2a, 2b, and 4 were subsequently developed by 3 researchers (N.B., A.H., P.M.), and revised through a 2-round Delphi consensus process. A final consensus teleconference resolved outstanding disagreements and clarified wording for checklist items. RESULTS: Sixty-one participants completed the initial survey, a clear majority indicating that new reporting guidelines were needed for IDEAL Stage 1 (69.5%), Stage 2a (78%), Stage 2b (74.6%), and Stage 4 (66%). A proposed set of checklists was modified by survey participants in 2 online Delphi rounds (n = 54 and n = 47, respectively), resulting in a penultimate checklist for each stage. Fourteen expert working group members finalized the checklist items and successfully resolved any outstanding areas without agreement on a consensus call. CONCLUSIONS: Participants familiar with IDEAL called for reporting guidelines for studies in all IDEAL stages except stage 3. The checklists developed have the potential to improve standards of reporting and thereby advance the quality of research on surgery and complex interventions and technologies, but require further evaluation in use.

Identifying competing interest disclosures in systematic reviews of surgical interventions and devices: a cross-sectional survey.

BACKGROUND: A competing interest is an important source of bias in research and disclosure is frequently employed as a strategy to manage it. Considering the importance of systematic reviews (SRs) and the varying prevalence of competing interests in different research fields, we conducted a survey to identify the range of competing interests in SRs assessing surgical interventions or devices and explored the association between the competing interest disclosures and authors' conclusions. METHODS: We retrieved SRs of surgical interventions and devices published in 2017 via PubMed. Information regarding general characteristics, funding sources, and competing interest disclosures were extracted. We conducted a descriptive analysis of the studies' characteristics and compared them between Cochrane SRs (CSRs) and non-Cochrane SRs using the Chi-square test. Results were expressed as odds ratio and their 95% confidence interval. RESULTS: One hundred fifty-five SRs published in 2017 were included in the study. More than half of the SRs (58.7%) reported their funding sources and 94.2% reported authors' competing interest disclosures. Among 146 SRs that stated competing interest disclosures, only 35 (22.6%) SRs declared at least one author had a competing interest. More than 40 terms were used to describe competing interests. Cochrane SRs (CSRs) were more likely to provide a detailed description of competing interests compared to those in non-CSRs (48.0% versus 25.4%, P = 0.023). No association between positive conclusions and competing interest disclosures was found (P = 0.484, OR = 0.43, 95%CI: 0.08, 2.16). In the subgroup analyses, SRs stating no competing interest disclosure were more likely to report positive conclusions than those stating at least one type of competing interest, but the difference is not significantly different (P = 0.406, OR = 1.38, 95%CI: 0.64, 2.98). CONCLUSION: In surgical SRs, there is a high percentage of competing interest disclosures but without detailed information. The identification and statement of competing interests with a detailed description, particularly the non-financial ones, needs improvement. Some efficient and effective methods/tools for identifying, quantifying, and minimizing potential competing interests in systematic reviews remains valuable.

The IDEAL framework in neurosurgery: a bibliometric analysis.

BACKGROUND: The Idea, Development, Exploration, Assessment and Long-term study (IDEAL) framework was created to provide a structured way for assessing and evaluating novel surgical techniques and devices. OBJECTIVES: The aim of this paper was to investigate the utilization of the IDEAL framework within neurosurgery, and to identify factors influencing implementation. METHODS: A bibliometric analysis of the 7 key IDEAL papers on Scopus, PubMed, Embase, Web of Science, and Google Scholar databases (2009-2019) was performed. A second journal-specific search then identified additional papers citing the IDEAL framework. Publications identified were screened by two independent reviewers to select neurosurgery-specific articles. RESULTS: The citation search identified 1336 articles. The journal search identified another 16 articles. Following deduplication and review, 51 relevant articles remained; 14 primary papers (27%) and 37 secondary papers (73%). Of the primary papers, 5 (36%) papers applied the IDEAL framework to their research correctly; two were aligned to the pre-IDEAL stage, one to the Idea and Development stages, and two to the Exploration stage. Of the secondary papers, 21 (57%) explicitly discussed the IDEAL framework. Eighteen (86%) of these were supportive of implementing the framework, while one was not, and two were neutral. CONCLUSION: The adoption of the IDEAL framework in neurosurgery has been slow, particularly for early-stage neurosurgical techniques and inventions. However, the largely positive reviews in secondary literature suggest potential for increased use that may be achieved with education and publicity.

No Surgical Innovation Without Evaluation: Evolution and Further Development of the IDEAL Framework and Recommendations.

OBJECTIVE: To update, clarify, and extend IDEAL concepts and recommendations. BACKGROUND: New surgical procedures, devices, and other complex interventions need robust evaluation for safety, efficacy, and effectiveness. Unlike new medicines, there is no internationally agreed evaluation pathway for generating and analyzing data throughout the life cycle of surgical innovations. The IDEAL Framework and Recommendations were designed to provide this pathway and they have been used increasingly since their introduction in 2009. Based on a Delphi survey, expert workshop and major discussions during IDEAL conferences held in Oxford (2016) and New York (2017), this article updates and extends the IDEAL Recommendations, identifies areas for future research, and discusses the ethical problems faced by investigators at each IDEAL stage. METHODS: The IDEAL Framework describes 5 stages of evolution for new surgical therapeutic interventions-Idea, Development, Exploration, Assessment, and Long-term Study. This comprehensive update proposes several modifications. First, a "Pre-IDEAL" stage describing preclinical studies has been added. Second we discuss potential adaptations to expand the scope of IDEAL (originally designed for surgical procedures) to accommodate therapeutic devices, through an IDEAL-D variant. Third, we explicitly recognise the value of comprehensive data collection through registries at all stages in the Framework and fourth, we examine the ethical issues that arise at each stage of IDEAL and underpin the recommendations. The Recommendations for each stage are reviewed, clarified and additional detail added. CONCLUSIONS: The intention of this article is to widen the practical use of IDEAL by clarifying the rationale for and practical details of the Recommendations. Additional research based on the experience of implementing these Recommendations is needed to further improve them.

The methodological quality of surgical randomized controlled trials: A cross-sectional systemic review.

Systematic reviews of RCTs have been developed to address end-users' needs and are regarded as the highest level of evidence. Flaws in the design, conduct and analyses of a systematic review can lead to erroneous conclusions and increase the research waste. We undertook a cross-sectional survey to identify the critical areas of weakness in systematic reviews for surgical interventions by AMSTAR 2. We searched PubMed, EMbase and Cochrane Library to summarize systematic reviews of surgical RCTs published in 2017. The information regarding general characteristics and methodological characteristics were gathered. We conducted descriptive analyses of study characteristics of included systematic reviews and explored the difference among varied methodological quality. Totally 141 systematic reviews were identified. We found only four reviews (2.8%) were high quality, 3 (2.1%) were moderate quality, 8 (5.7%) were low quality, and the remaining 126 (89.4%) were of critical low quality. The critical weaknesses were lack of pre-registration or published protocols (29.1%), comprehensive literature search (17.7%), lists of excluded studies and reasons for exclusion (19.1%), description of detailed interventions (8.5%), extraction of funding source from included trials (10.6%), and consideration of the risk of bias of included trials when synthesized (16.3%) and interpret (20.6%) the results. Higher methodological quality was only positively associated with Cochrane systematic review. Although two-thirds of included systematic reviews in the field of surgery were published in journals ranking Q1, the methodological quality is suboptimal and needs to be substantially improved. More efforts of multi-disciplinary teams' collaboration, continual education and training, integrally connection between primary studies and systematic review and contributing surgical research to practice should be imperative.

Identifying research waste from surgical research: a protocol for assessing compliance with the IDEAL framework and recommendations.

INTRODUCTION: Approximately £1130 billion was invested in research worldwide in 2016, and 9.6% of this was on biomedical research. However, about 85% of biomedical research investment is wasted. The Lancet published a series to identify five categories relating to research waste and in 2014. Some categories of research waste in surgery are avoidable by complying with the Idea, Development, Exploration, Assessment, Long-term follow-up (IDEAL) framework for it enables researchers to design, conduct and report surgical studies robustly and transparently. This review aims to examine the extent to which surgical studies adhered to the IDEAL framework and estimate the amount of overall research waste that could be avoided if compliance was improved. METHODS: We will search for potential studies published in English and between 1 January 2018 and 31 December 2018 via PubMed. Teams of paired reviewers will screen titles, abstracts and full texts independently. Two researchers will extract data from each paper. Data will be collected about general information and specialised information in each stage, and our IDEAL Compliance Appraisal tool will be used to analyse included studies. Descriptive statistics and χ2 or Fisher's exact tests for comparisons will be presented. DISCUSSION: Our study will provide important information about whether compliance with the specific IDEAL Recommendations has reduced research waste in surgical and therapeutic device studies. And we will identify particular key aspects that are worse and need to focus on improving those in future education.

Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network.

Although current electronic methods of scientific publishing offer increased opportunities for publishing all research studies and describing them in sufficient detail, health research literature still suffers from many shortcomings. These shortcomings seriously undermine the value and utility of the literature and waste scarce resources invested in the research. In recent years there have been several positive steps aimed at improving this situation, such as a strengthening of journals' policies on research publication and the wide requirement to register clinical trials.The EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network is an international initiative set up to advance high quality reporting of health research studies; it promotes good reporting practices including the wider implementation of reporting guidelines. EQUATOR provides free online resources http://www.equator-network.org supported by education and training activities and assists in the development of robust reporting guidelines. This paper outlines EQUATOR's goals and activities and offers suggestions for organizations and individuals involved in health research on how to strengthen research reporting.

Evolution of transanal total mesorectal excision according to the IDEAL framework.

Tremendous innovations have taken place in surgical procedures, but contrary to drug development, this process has been unregulated in the past. IDEAL promotes a structured framework for the safe implementation and assessment of a new surgical technique or intervention, by describing five stages for evaluating and reporting of innovations: Idea, Development, Exploration, Assessment and Long term. Transanal total mesorectal excision (TaTME) is a relatively new technique in rectal cancer surgery that has attracted huge interest and increasing adoption worldwide. This review article aims to provide an overview of the evolution of TaTME, according to the IDEAL framework, which guides us in the difficult yet exciting process of surgical innovation.

A protocol for the development of reporting guidelines for IDEAL stage studies.

BACKGROUND: New surgical procedures, devices and other complex interventions need robust evaluation for safety, efficacy and effectiveness. The IDEAL Framework and Recommendations lay out a pathway to achieve this and offer general guidance on how studies at each stage should be reported. However, researchers require some assistance in translating theory into practice. We will develop a set of reporting guidelines for each IDEAL stage where deemed necessary through Delphi consensus methodology. METHODS: For each IDEAL stage requiring a new set of reporting guidelines, we will use the following process. We will search for the relevant reporting guidelines already in existence and use principles developed by the IDEAL Collaboration to compile the initial long list of potential checklist items. In each round, the participants will rate the importance of reporting each element on a nine-point Likert scale as proposed by the GRADE group. Sequential rounds and questionnaire administration and completion will take place until a final set of items is produced. There will then be a final consensus meeting of a working group to condense and refine the final recommendations for the reporting guidelines.

Appraising the uptake and use of the IDEAL Framework and Recommendations: A review of the literature.

INTRODUCTION: Evaluation of new surgical innovations is complex and variably regulated, and historically the quality of surgical studies has been criticized. The IDEAL (Idea, Development, Exploration, Assessment, Long-term monitoring) Framework was established to provide a pathway for evaluating surgical innovations at each stage of their development in order to produce high quality surgical research. Since the inception of IDEAL in 2009, there has been no assessment of its use. In this review, we look at the uptake and usage of IDEAL by examining the published literature. METHODS: We conducted a literature search to identify all of the publications that cited IDEAL and included only those papers that intentionally used IDEAL as part of the study methodology. We then characterized these publications by year of publication, specialty, and geographical location. We performed a critical appraisal of Stage 1, 2a, and 2b studies in order to assess the degree to which authors have correctly followed the Framework and Recommendations. RESULTS: We found 790 citations of IDEAL publications, and after abstract and full-text screening, 38 prospective studies for a surgical innovation that used IDEAL remained. We saw an overall increase in the uptake of IDEAL, with a predominance in urology and origin in the United Kingdom. The critical appraisal showed that although authors identified their project as using IDEAL, they often failed to include key IDEAL characteristics; this was especially true for the features unique to IDEAL Stages 2a and 2b. CONCLUSION: It is evident from the large number of studies citing IDEAL that the importance and challenges of reporting surgical research is well recognized among researchers. There is growing enthusiasm for using IDEAL but the current level of understanding of the Recommendations is low. Clearer and more comprehensive explanation of the application of the IDEAL Framework and Recommendations is needed to guide surgical researchers undertaking IDEAL based studies of surgical innovations.

Evidence-Based Evaluation of Practice and Innovation in Physical Therapy Using the IDEAL-Physio Framework.

The IDEAL framework is an established method for initial and ongoing evaluations of innovation and practice for complex health care interventions. First derived for surgical sciences and embedded at a global level for evaluating surgery/surgical devices, the IDEAL framework is based on the principle that innovation and evaluation in clinical practice can, and should, evolve together in an ordered manner: from conception to development and then to validation by appropriate clinical studies and, finally, longer-term follow-up. This framework is highly suited to other complex, nonpharmacological interventions, such as physical therapist interventions. This perspective outlines the application of IDEAL to physical therapy in the new IDEAL-Physio framework. The IDEAL-Physio framework comprises 5 stages. In stage 1, the idea phase, formal data collection should begin. Stage 2a is the phase for iterative improvement and adjustment with thorough data recording. Stage 2b involves the onset of formal evaluation using systematically collected group or cohort data. Stage 3 is the phase for formal comparative assessment of treatment, usually involving randomized studies. Stage 4 involves long-term follow-up. The IDEAL-Physio framework is recommended as a method for guiding and evaluating both innovation and practice in physical therapy, with the overall goal of providing better evidence-based care.

Adapting the IDEAL Framework and Recommendations for medical device evaluation: A modified Delphi survey.

INTRODUCTION: Current regulatory systems for medical device marketing approval lack adequate requirements for evidence of safety and efficacy. The Total Product Life Cycle (TPLC) concept, with clinical use and marketing expanding as evidence develops, has won support, but lacks a template to define evidence requirements at different stages. The IDEAL Framework & Recommendations, originally developed for new surgical procedures, might provide such a template, but may require modification. METHODS: We conducted a Delphi expert consensus exercise to determine how IDEAL might be modified to accommodate the needs of device regulation. 34 experts were invited to participate in 3 rounds of questioning, with feedback of the results of each round to participants before the next. RESULTS: 27 of 34 experts responded in at least one survey round. Experts agreed that, after appropriate modifications, IDEAL could form an evidence template for a TPLC-based regulatory system. Necessary modifications include a new Stage 0 should guide reporting of pre-clinical studies, expansion of registries to all stages, and omission of IDEAL stages 2 and 3 for "successor" devices under certain conditions. DISCUSSION: A standard approach to TPLC evaluation of medical devices does not currently exist. The IDEAL Framework, if modified appropriately, could fill such a void and improve the safety of new medical devices.

The methodological framework of surgical innovation: the introduction of IDEAL framework and recommendation / 中国胸心血管外科临床杂志

@#Surgical innovation is an important part of surgical research and practice. The evaluation of surgical innovation through the stages is similar to those for drug development, but with important differences. The Idea, Development, Exploration, Assessment, and Long-term follow-up (IDEAL) Framework and Recommendations represent a new paradigm for the evaluation of surgical intervention and devices which was developed in 2009. The IDEAL is a five-stage framework involving the nature stages of surgical innovation, together with recommendations for surgical research pathway. The Framework and Recommendations were updated and published in 2019, which added a pre-IDEAL stage if necessary. The updated IDEAL also underlines the purpose, key question and ethical issues for each stage. In the first paper of IDEAL Framework and Recommendations series, we conducted a comprehensive introduction of IDEAL (e.g. the development, updates and application of IDEAL) to promote the dissemination and application of IDEAL in China.

The methodological framework of surgical innovation: The interpretation of IDEAL reporting guideline / 中国胸心血管外科临床杂志

@#Adherence to reporting guidelines contributes to report methodology and outcomes of research distinctly and transparently. There are some checklists with specific study types related to surgery on the EQUATOR Network’s website. However, the IDEAL framework focuses on stepwise evaluation of surgical innovation through all stages with some key elements, which those existing guidelines may not mention. This likely results in the inaccuracy in reporting in studies attempting to follow the IDEAL recommendations and suggests a pressing need for IDEAL reporting guidelines. Considering these limitations, the IDEAL developed the IDEAL reporting guidelines between October 2018 and May 2019. The paper aimed to provide interpretation of IDEAL reporting guideline, and promote its understanding and use among Chinese researchers.

Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review.

OBJECTIVE: To assess whether the completeness of reporting of health research is related to journals' endorsement of reporting guidelines. DESIGN: Systematic review. DATA SOURCES: Reporting guidelines from a published systematic review and the EQUATOR Network (October 2011). Studies assessing the completeness of reporting by using an included reporting guideline (termed "evaluations") (1990 to October 2011; addendum searches in January 2012) from searches of either Medline, Embase, and the Cochrane Methodology Register or Scopus, depending on reporting guideline name. STUDY SELECTION: English language reporting guidelines that provided explicit guidance for reporting, described the guidance development process, and indicated use of a consensus development process were included. The CONSORT statement was excluded, as evaluations of adherence to CONSORT had previously been reviewed. English or French language evaluations of included reporting guidelines were eligible if they assessed the completeness of reporting of studies as a primary intent and those included studies enabled the comparisons of interest (that is, after versus before journal endorsement and/or endorsing versus non-endorsing journals). DATA EXTRACTION: Potentially eligible evaluations of included guidelines were screened initially by title and abstract and then as full text reports. If eligibility was unclear, authors of evaluations were contacted; journals' websites were consulted for endorsement information where needed. The completeness of reporting of reporting guidelines was analyzed in relation to endorsement by item and, where consistent with the authors' analysis, a mean summed score. RESULTS: 101 reporting guidelines were included. Of 15,249 records retrieved from the search for evaluations, 26 evaluations that assessed completeness of reporting in relation to endorsement for nine reporting guidelines were identified. Of those, 13 evaluations assessing seven reporting guidelines (BMJ economic checklist, CONSORT for harms, PRISMA, QUOROM, STARD, STRICTA, and STROBE) could be analyzed. Reporting guideline items were assessed by few evaluations. CONCLUSIONS: The completeness of reporting of only nine of 101 health research reporting guidelines (excluding CONSORT) has been evaluated in relation to journals' endorsement. Items from seven reporting guidelines were quantitatively analyzed, by few evaluations each. Insufficient evidence exists to determine the relation between journals' endorsement of reporting guidelines and the completeness of reporting of published health research reports. Journal editors and researchers should consider collaborative prospectively designed, controlled studies to provide more robust evidence. SYSTEMATIC REVIEW REGISTRATION: Not registered; no known register currently accepts protocols for methodology systematic reviews.

Reporting characteristics of non-primary publications of results of randomized trials: a cross-sectional review.

BACKGROUND: For a randomized trial, the primary publication is usually the one which reports the results of the primary outcome and provides consolidated data from all study centers. Other aspects of a randomized trial's findings (that is, non-primary results) are often reported in subsequent publications. METHODS: We carried out a cross-sectional review of the characteristics and type of information reported in non-primary reports (n = 69) of randomized trials (indexed in PubMed core clinical journals in 2009) and whether they report pre-specified or exploratory analyses. We also compared consistency of information in non-primary publications with that reported in the primary publication. RESULTS: The majority (n = 56; 81%) of non-primary publications were large, multicenter trials, published in specialty journals. Most reported subgroup analyses (n = 27; 39%), analyzing a specific subgroup of patients from the randomized trial, or reported on secondary outcomes (n = 29; 42%); 19% (n = 13) reported extended follow-up. Less than half reported details of trial registration (n = 30; 43%) or the trial protocol (n = 27; 39%) and in 41% (n = 28) it was unclear from reading the abstract that the report was not the primary publication for the trial. Non-primary publications often analyzed and reported multiple different outcomes (16% reported >20 outcomes) and in 10% (n = 7) it was unclear how many outcomes had actually been assessed; in 42% (n = 29) it was unclear whether the analyses reported were pre-specified or exploratory. Only 39% (n = 27) of non-primary publications described the primary outcome of the randomized trial, 6% (n = 4) reported its numerical results and 9% (n = 6) details of how participants were randomized. CONCLUSION: Non-primary publications often lack important information about the randomized trial and the type of analyses conducted and whether these were pre-specified or exploratory to enable readers to accurately identify and assess the validity and reliably of the study findings. We provide recommendations for what information authors should include in non-primary reports of randomized trials.

Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals.

BACKGROUND: Pre-publication peer review of manuscripts should enhance the value of research publications to readers who may wish to utilize findings in clinical care or health policy-making. Much published research across all medical specialties is not useful, may be misleading, wasteful and even harmful. Reporting guidelines are tools that in addition to helping authors prepare better manuscripts may help peer reviewers in assessing them. We examined journals' instructions to peer reviewers to see if and how reviewers are encouraged to use them. METHODS: We surveyed websites of 116 journals from the McMaster list. Main outcomes were 1) identification of online instructions to peer reviewers and 2) presence or absence of key domains within instructions: on journal logistics, reviewer etiquette and addressing manuscript content (11 domains). FINDINGS: Only 41/116 journals (35%) provided online instructions. All 41 guided reviewers about the logistics of their review processes, 38 (93%) outlined standards of behaviour expected and 39 (95%) contained instruction about evaluating the manuscript content. There was great variation in explicit instruction for reviewers about how to evaluate manuscript content. Almost half of the online instructions 19/41 (46%) mentioned reporting guidelines usually as general statements suggesting they may be useful or asking whether authors had followed them rather than clear instructions about how to use them. All 19 named CONSORT for reporting randomized trials but there was little mention of CONSORT extensions. PRISMA, QUOROM (forerunner of PRISMA), STARD, STROBE and MOOSE were mentioned by several journals. No other reporting guideline was mentioned by more than two journals. CONCLUSIONS: Although almost half of instructions mentioned reporting guidelines, their value in improving research publications is not being fully realised. Journals have a responsibility to support peer reviewers. We make several recommendations including wider reference to the EQUATOR Network online library (www.equator-network.org/).