No frugal innovation without frugal evaluation: the Global IDEAL Sub-Framework.

Objective: The Global IDEAL Sub-Framework Study aimed to combine the intended effects of the 2009/2019 IDEAL (Idea, Development, Exploration, Assessment, Long-term study) Framework recommendations on evaluating surgical innovation with the vision outlined by the 2015 Lancet Commission on Global Surgery to provide recommendations for evaluating surgical innovation in low-resource environments. Design: A mixture of methods including an online global survey and semistructured interviews (SSIs). Quantitative data were summarized with descriptive statistics and qualitative data were analyzed using the Framework Method. Participants: Surgeons and surgical researchers from any country. Main outcome measures: Findings were used to suggest the nature of adaptations to the IDEAL Framework to address the particular problems of evaluation in low-resource settings. Results: The online survey yielded 66 responses representing experience from 40 countries, and nine individual SSIs were conducted. Most respondents (n=49; 74.2%) had experience evaluating surgical technologies across a range of life cycle stages. Innovation was most frequently adopted based on colleague recommendation or clinical evaluation in other countries. Four themes emerged, centered around: frugal innovation in technological development; evaluating the same technology/innovation in different contexts; additional methodologies important in evaluation of surgical innovation in low/middle-income countries; and support for low-income country researchers along the evaluation pathway. Conclusions: The Global IDEAL Sub-Framework provides suggestions for modified IDEAL recommendations aimed at dealing with the special problems found in this setting. These will require validation in a stakeholder consensus forum, and qualitative assessment in pilot studies. From assisting researchers with identification of the correct evaluation stage, to providing context-specific recommendations relevant to the whole evaluation pathway, this process will aim to develop a comprehensive and applicable set of guidance that will benefit surgical innovation and patients globally.

Holistic Human-Serving Digitization of Health Care Needs Integrated Automated System-Level Assessment Tools.

Digital health tools, platforms, and artificial intelligence- or machine learning-based clinical decision support systems are increasingly part of health delivery approaches, with an ever-greater degree of system interaction. Critical to the successful deployment of these tools is their functional integration into existing clinical routines and workflows. This depends on system interoperability and on intuitive and safe user interface design. The importance of minimizing emergent workflow stress through human factors research and purposeful design for integration cannot be overstated. Usability of tools in practice is as important as algorithm quality. Regulatory and health technology assessment frameworks recognize the importance of these factors to a certain extent, but their focus remains mainly on the individual product rather than on emergent system and workflow effects. The measurement of performance and user experience has so far been performed in ad hoc, nonstandardized ways by individual actors using their own evaluation approaches. We propose that a standard framework for system-level and holistic evaluation could be built into interacting digital systems to enable systematic and standardized system-wide, multiproduct, postmarket surveillance and technology assessment. Such a system could be made available to developers through regulatory or assessment bodies as an application programming interface and could be a requirement for digital tool certification, just as interoperability is. This would enable health systems and tool developers to collect system-level data directly from real device use cases, enabling the controlled and safe delivery of systematic quality assessment or improvement studies suitable for the complexity and interconnectedness of clinical workflows using developing digital health technologies.

Very large treatment effects in randomised trials as an empirical marker to indicate whether subsequent trials are necessary: meta-epidemiological assessment.

OBJECTIVE:  To examine whether a very large effect (VLE; defined as a relative risk of ≤0.2 or ≥5) in a randomised trial could be an empirical marker that subsequent trials are unnecessary. DESIGN:  Meta-epidemiological assessment of existing published data on randomised trials. DATA SOURCES:  Cochrane Database of Systematic Reviews (2010, issue 7) with data on subsequent large trials updated to 2015, issue 12. ELIGIBILITY CRITERIA:  All binary outcome forest plots were selected, which contained an index randomised trial with a VLE that was nominally statistically significant (P<0.05), included a subsequent large randomised trial (≥200 events and ≥200 non-events) for validation of the effect, assessed a primary outcome of the review, and was not a subgroup or sensitivity analysis. RESULTS:  Of 3082 reviews yielding 85 002 forest plots, only 44 (0.05%) satisfied the inclusion criteria. Index trials were generally small, with a median sample of 99 (median 14 events). Few index trials were rated at low risk of bias (9 of 44; 20%). The relative risk was closer to the null in the subsequent large trials in 43 of 44 cases. Subsequent large trial data failed to find a statistically significant (P<0.05) effect in the same direction in 19 cases (43%, 95% confidence interval 29% to 58%). Even when the subsequent large trials did find a significant effect in the same direction, the additional primary outcomes in most of these trials would have to be considered before deciding in favour of using the intervention. Subsequent large trial data found a statistically significant effect in the same direction in 19 of 21 cases when the index trial also had a value of P<0.001. CONCLUSIONS:  The frequency of VLEs followed by a large trial is vanishingly small, and where they occur they do not appear to be a reliable marker for a benefit that is reproducible and directly actionable. An empirical rule using a VLE in a randomised controlled trial as a marker that further trials are unnecessary would be neither practical nor useful. Caution should be taken when interpreting small studies with very large treatment effects.

Human factors in critical care: towards standardized integrated human-centred systems of work.

PURPOSE OF REVIEW: Improvements in safety and quality benefit from a systems approach. Human factors is the study and practice of the relationship between humans and systems. This review examines recent advances in human factors in healthcare. RECENT FINDINGS: Early studies focused on understanding incidents, and on the translation of principles from aviation to healthcare, which demonstrated a useful but limited application of the human factors approach. More recent studies have begun to address the complexity of the relationship between human behaviour and technology, tasks, environment and organization. Human factors frameworks have been usefully applied that aid in these complex considerations, providing a better understanding of the healthcare system, and a much broader range of solutions to problems than checklists, protocols or training. In particular, in improving equipment design and procurement; improving job design by understanding the demands and tasks of individual healthcare practitioners; in improving what and when training is delivered; and the integration of these complex system components into a coherent whole. SUMMARY: The human factors approach is not yet mature in healthcare, but the importance is being increasingly recognized, and the breadth of application continually expanded.

Evaluation and stages of surgical innovations.

Surgical innovation is an important part of surgical practice. Its assessment is complex because of idiosyncrasies related to surgical practice, but necessary so that introduction and adoption of surgical innovations can derive from evidence-based principles rather than trial and error. A regulatory framework is also desirable to protect patients against the potential harms of any novel procedure. In this first of three Series papers on surgical innovation and evaluation, we propose a five-stage paradigm to describe the development of innovative surgical procedures.

No surgical innovation without evaluation: the IDEAL recommendations.

Surgery and other invasive therapies are complex interventions, the assessment of which is challenged by factors that depend on operator, team, and setting, such as learning curves, quality variations, and perception of equipoise. We propose recommendations for the assessment of surgery based on a five-stage description of the surgical development process. We also encourage the widespread use of prospective databases and registries. Reports of new techniques should be registered as a professional duty, anonymously if necessary when outcomes are adverse. Case series studies should be replaced by prospective development studies for early technical modifications and by prospective research databases for later pre-trial evaluation. Protocols for these studies should be registered publicly. Statistical process control techniques can be useful in both early and late assessment. Randomised trials should be used whenever possible to investigate efficacy, but adequate pre-trial data are essential to allow power calculations, clarify the definition and indications of the intervention, and develop quality measures. Difficulties in doing randomised clinical trials should be addressed by measures to evaluate learning curves and alleviate equipoise problems. Alternative prospective designs, such as interrupted time series studies, should be used when randomised trials are not feasible. Established procedures should be monitored with prospective databases to analyse outcome variations and to identify late and rare events. Achievement of improved design, conduct, and reporting of surgical research will need concerted action by editors, funders of health care and research, regulatory bodies, and professional societies.

Challenges in evaluating surgical innovation.

Research on surgical interventions is associated with several methodological and practical challenges of which few, if any, apply only to surgery. However, surgical evaluation is especially demanding because many of these challenges coincide. In this report, the second of three on surgical innovation and evaluation, we discuss obstacles related to the study design of randomised controlled trials and non-randomised studies assessing surgical interventions. We also describe the issues related to the nature of surgical procedures-for example, their complexity, surgeon-related factors, and the range of outcomes. Although difficult, surgical evaluation is achievable and necessary. Solutions tailored to surgical research and a framework for generating evidence on which to base surgical practice are essential.

Gentronix.

Gentronix is an innovative biotechnology company with the aim of providing cell-based assays and systems to ease or remove some of the current bottlenecks in preclinical drug discovery and development. Gentronix's first product, GreenScreen trade mark GC, is a combined genotoxicity and cytotoxicity test designed for high-throughput screening of new pharmaceutical lead compounds, giving drug development scientists a cost-effective preview of the battery of mandatory regulatory genotoxicity tests. Future products based on the company's extensive intellectual property rights portfolio will combine the company's expertise in cellular biosensor systems with novel optical detection technology to deliver a range of products for other applications within drug discovery, as well as industrial and environmental monitoring of toxic chemicals.