Modelling clinical narrative as computable knowledge: The NICE computable implementation guidance project.

Introduction: Translating narrative clinical guidelines to computable knowledge is a long-standing challenge that has seen a diverse range of approaches. The UK National Institute for Health and Care Excellence (NICE) Content Advisory Board (CAB) aims ultimately to (1) guide clinical decision support and other software developers to increase traceability, fidelity and consistency in supporting clinical use of NICE recommendations, (2) guide local practice audit and intervention to reduce unwarranted variation, (3) provide feedback to NICE on how future recommendations should be developed. Objectives: The first phase of work was to explore a range of technical approaches to transition NICE toward the production of natively digital content. Methods: Following an initial 'collaborathon' in November 2022, the NICE Computable Implementation Guidance project (NCIG) was established. We held a series of workstream calls approximately fortnightly, focusing on (1) user stories and trigger events, (2) information model and definitions, (3) horizon-scanning and output format. A second collaborathon was held in March 2023 to consolidate progress across the workstreams and agree residual actions to complete. Results: While we initially focussed on technical implementation standards, we decided that an intermediate logical model was a more achievable first step in the journey from narrative to fully computable representation. NCIG adopted the WHO Digital Adaptation Kit (DAK) as a technology-agnostic method to model user scenarios, personae, processes and workflow, core data elements and decision-support logic. Further work will address indicators, such as prescribing compliance, and implementation in document templates for primary care patient record systems. Conclusions: The project has shown that the WHO DAK, with some modification, is a promising approach to build technology-neutral logical specifications of NICE recommendations. Implementation of concurrent computable modelling by multidisciplinary teams during guideline development poses methodological and cultural questions that are complex but tractable given suitable will and leadership.

Which computable biomedical knowledge objects will be regulated? Results of a UK workshop discussing the regulation of knowledge libraries and software as a medical device.

Introduction: To understand when knowledge objects in a computable biomedical knowledge library are likely to be subject to regulation as a medical device in the United Kingdom. Methods: A briefing paper was circulated to a multi-disciplinary group of 25 including regulators, lawyers and others with insights into device regulation. A 1-day workshop was convened to discuss questions relating to our aim. A discussion paper was drafted by lead authors and circulated to other authors for their comments and contributions. Results: This article reports on those deliberations and describes how UK device regulators are likely to treat the different kinds of knowledge objects that may be stored in computable biomedical knowledge libraries. While our focus is the likely approach of UK regulators, our analogies and analysis will also be relevant to the approaches taken by regulators elsewhere. We include a table examining the implications for each of the four knowledge levels described by Boxwala in 2011 and propose an additional level. Conclusions: If a knowledge object is described as directly executable for a medical purpose to provide decision support, it will generally be in scope of UK regulation as "software as a medical device." However, if the knowledge object consists of an algorithm, a ruleset, pseudocode or some other representation that is not directly executable and whose developers make no claim that it can be used for a medical purpose, it is not likely to be subject to regulation. We expect similar reasoning to be applied by regulators in other countries.

Artificial intelligence and clinical decision support: clinicians' perspectives on trust, trustworthiness, and liability.

Artificial intelligence (AI) could revolutionise health care, potentially improving clinician decision making and patient safety, and reducing the impact of workforce shortages. However, policymakers and regulators have concerns over whether AI and clinical decision support systems (CDSSs) are trusted by stakeholders, and indeed whether they are worthy of trust. Yet, what is meant by trust and trustworthiness is often implicit, and it may not be clear who or what is being trusted. We address these lacunae, focusing largely on the perspective(s) of clinicians on trust and trustworthiness in AI and CDSSs. Empirical studies suggest that clinicians' concerns about their use include the accuracy of advice given and potential legal liability if harm to a patient occurs. Onora O'Neill's conceptualisation of trust and trustworthiness provides the framework for our analysis, generating a productive understanding of clinicians' reported trust issues. Through unpacking these concepts, we gain greater clarity over the meaning ascribed to them by stakeholders; delimit the extent to which stakeholders are talking at cross purposes; and promote the continued utility of trust and trustworthiness as useful concepts in current debates around the use of AI and CDSSs.

Insights from developing and evaluating the NHS blood choices transfusion app to support junior and middle-grade doctor decision making against guidelines.

OBJECTIVES: To: 1. Develop a CE-marked smartphone App to support doctors' concordance with transfusion guidelines in non-bleeding adult patients, emphasising informed consent and anaemia management. 2. Test App accuracy and potential to improve user decisions. BACKGROUND: Studies have shown inappropriate use of blood components and that most junior doctors own smartphones with medical apps. METHODS: A multidisciplinary team developed App screens and logic through an iterative process based on national guidelines. Thirty medical or surgical transfusion scenarios were developed based on national guidelines and each sent to Consultant Haematologist experts in Transfusion Medicine. To obtain a clinical consensus and exclude ambiguous scenarios, their independent decisions and associated certainty were compared. The consensus clinical decision was then compared with guidance from the App. To explore potential App impact on simulated user decisions, 26 junior doctors responded to five transfusion scenarios before and after access to the App. RESULTS: The Blood Choices App agreed with 91% (95% CI: 72%-99%) of expert decisions with a sensitivity of 100% (69% to 100%) and specificity of 85% (55%-98%). Excluding one malfunction scenario, the App had the potential to increase correct decisions by junior doctors from 83% (73%-90%) pre-App use to 96% (88%-99%) post (p-value 0.013), with 90% (67%-99%) saying they would use it in practice. CONCLUSIONS: Transfusion guidelines can be converted into an App with potential to improve guideline concordance. However, evaluating such Apps is essential to understand their limitations, detect malfunctions and prevent harm.

An overview of the National COVID-19 Chest Imaging Database: data quality and cohort analysis.

BACKGROUND: The National COVID-19 Chest Imaging Database (NCCID) is a centralized database containing mainly chest X-rays and computed tomography scans from patients across the UK. The objective of the initiative is to support a better understanding of the coronavirus SARS-CoV-2 disease (COVID-19) and the development of machine learning technologies that will improve care for patients hospitalized with a severe COVID-19 infection. This article introduces the training dataset, including a snapshot analysis covering the completeness of clinical data, and availability of image data for the various use-cases (diagnosis, prognosis, longitudinal risk). An additional cohort analysis measures how well the NCCID represents the wider COVID-19-affected UK population in terms of geographic, demographic, and temporal coverage. FINDINGS: The NCCID offers high-quality DICOM images acquired across a variety of imaging machinery; multiple time points including historical images are available for a subset of patients. This volume and variety make the database well suited to development of diagnostic/prognostic models for COVID-associated respiratory conditions. Historical images and clinical data may aid long-term risk stratification, particularly as availability of comorbidity data increases through linkage to other resources. The cohort analysis revealed good alignment to general UK COVID-19 statistics for some categories, e.g., sex, whilst identifying areas for improvements to data collection methods, particularly geographic coverage. CONCLUSION: The NCCID is a growing resource that provides researchers with a large, high-quality database that can be leveraged both to support the response to the COVID-19 pandemic and as a test bed for building clinically viable medical imaging models.

Correction: Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study.

[This corrects the article DOI: .].

Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study.

BACKGROUND: Despite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase. OBJECTIVE: The objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase. METHODS: Three consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an "eHealth evaluation cycle" and subsequently compose the online "eHealth methodology guide." RESULTS: The scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an "eHealth evaluation cycle" was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the "eHealth methodology guide" was composed by assigning the 75 evaluation approaches to the specific study phases of the "eHealth evaluation cycle." CONCLUSIONS: Seventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online "eHealth methodology guide." By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the "eHealth evaluation cycle," the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

Digital tools for the recruitment and retention of participants in randomised controlled trials: a systematic map.

BACKGROUND: Recruiting and retaining participants in randomised controlled trials (RCTs) is challenging. Digital tools, such as social media, data mining, email or text-messaging, could improve recruitment or retention, but an overview of this research area is lacking. We aimed to systematically map the characteristics of digital recruitment and retention tools for RCTs, and the features of the comparative studies that have evaluated the effectiveness of these tools during the past 10 years. METHODS: We searched Medline, Embase, other databases, the Internet, and relevant web sites in July 2018 to identify comparative studies of digital tools for recruiting and/or retaining participants in health RCTs. Two reviewers independently screened references against protocol-specified eligibility criteria. Included studies were coded by one reviewer with 20% checked by a second reviewer, using pre-defined keywords to describe characteristics of the studies, populations and digital tools evaluated. RESULTS: We identified 9163 potentially relevant references, of which 104 articles reporting 105 comparative studies were included in the systematic map. The number of published studies on digital tools has doubled in the past decade, but most studies evaluated digital tools for recruitment rather than retention. The key health areas investigated were health promotion, cancers, circulatory system diseases and mental health. Few studies focussed on minority or under-served populations, and most studies were observational. The most frequently-studied digital tools were social media, Internet sites, email and tv/radio for recruitment; and email and text-messaging for retention. One quarter of the studies measured efficiency (cost per recruited or retained participant) but few studies have evaluated people's attitudes towards the use of digital tools. CONCLUSIONS: This systematic map highlights a number of evidence gaps and may help stakeholders to identify and prioritise further research needs. In particular, there is a need for rigorous research on the efficiency of the digital tools and their impact on RCT participants and investigators, perhaps as studies-within-a-trial (SWAT) research. There is also a need for research into how digital tools may improve participant retention in RCTs which is currently underrepresented relative to recruitment research. REGISTRATION: Not registered; based on a pre-specified protocol, peer-reviewed by the project's Advisory Board.

What do senior physicians think about AI and clinical decision support systems: Quantitative and qualitative analysis of data from specialty societies.

AIMS: The aim was to help physicians engage with NHS and other policymakers about the use, procurement and regulation of artificial intelligence, algorithms and clinical decision support systems (CDSS) in the NHS by identifying the professional benefits of and concerns about these systems. METHODS: We piloted a three-page survey instrument with closed and open-ended questions on SurveyMonkey, then circulated it to specialty societies via email. Both quantitative and qualitative methods were used to analyse responses. RESULTS: The results include the current usage of CDSS; identified benefits; concerns about quality; concerns about regulation, professional practice, ethics and liability, as well as actions being taken by the specialty societies to address these; and aspects of CDSS quality that need to be tested. CONCLUSION: While results confirm many expected benefits and concerns about CDSS, they raise new professional concerns and suggest further actions to explore with partners on behalf of the physician community.

Using digital tools in the recruitment and retention in randomised controlled trials: survey of UK Clinical Trial Units and a qualitative study.

BACKGROUND: Recruitment and retention of participants in randomised controlled trials (RCTs) is a key determinant of success but is challenging. Trialists and UK Clinical Research Collaboration (UKCRC) Clinical Trials Units (CTUs) are increasingly exploring the use of digital tools to identify, recruit and retain participants. The aim of this UK National Institute for Health Research (NIHR) study was to identify what digital tools are currently used by CTUs and understand the performance characteristics required to be judged useful. METHODS: A scoping of searches (and a survey with NIHR funding staff), a survey with all 52 UKCRC CTUs and 16 qualitative interviews were conducted with five stakeholder groups including trialists within CTUs, funders and research participants. A purposive sampling approach was used to conduct the qualitative interviews during March-June 2018. Qualitative data were analysed using a content analysis and inductive approach. RESULTS: Responses from 24 (46%) CTUs identified that database-screening tools were the most widely used digital tool for recruitment, with the majority being considered effective. The reason (and to whom) these tools were considered effective was in identifying potential participants (for both Site staff and CTU staff) and reaching recruitment target (for CTU staff/CI). Fewer retention tools were used, with short message service (SMS) or email reminders to participants being the most reported. The qualitative interviews revealed five themes across all groups: 'security and transparency'; 'inclusivity and engagement'; 'human interaction'; 'obstacles and risks'; and 'potential benefits'. There was a high level of stakeholder acceptance of the use of digital tools to support trials, despite the lack of evidence to support them over more traditional techniques. Certain differences and similarities between stakeholder groups demonstrated the complexity and challenges of using digital tools for recruiting and retaining research participants. CONCLUSIONS: Our studies identified a range of digital tools in use in recruitment and retention of RCTs, despite the lack of high-quality evidence to support their use. Understanding the type of digital tools in use to support recruitment and retention will help to inform funders and the wider research community about their value and relevance for future RCTs. Consideration of further focused digital tool reviews and primary research will help to reduce gaps in the evidence base.

Preserving the Open Access Benefits Pioneered by the Journal of Medical Internet Research and Discouraging Fraudulent Journals.

The Journal of Medical Internet Research (JMIR) was an early pioneer of open access online publishing, and two decades later, some readers and authors may have forgotten the challenges of previous scientific publishing models. This commentary summarizes the many advantages of open access publishing for each of the main stakeholders in scientific publishing and reminds us that, like every innovation, there are disadvantages that we need to guard against, such as the problem of fraudulent journals. This paper then reviews the potential impact of some current initiatives, such as Plan S and JMIRx, concluding with some suggestions to help new open-access publishers ensure that the advantages of open access publishing outweigh the challenges.

The Need for Theory to Inform Clinical Information Systems and Professionalise the Health Informatics Discipline.

This chapter introduces the idea of theories in health informatics, defines what we mean by theory and distinguishes theories from models, frameworks and predictive principles. After explaining why theories and predictive principles are needed to help us professionalize our discipline, the chapter offers five criteria for a successful predictive principle, discusses how to evaluate predictive principles and theories and links this with the emerging field of evidence-based health informatics. The chapter concludes with three actions needed to move the discipline of theory-based health informatics forward.

Health Behaviour Theory in Health Informatics: Support for Positive Change.

The rising use of the Internet and information technology has made computerized interventions an attractive channel for providing advice and support for behaviour change. Health behaviour and behaviour change theories are a family of theories which aim to explain the mechanisms by which human behaviours change and use that knowledge to promote change. Among the best-known of these theories are the Social Learning and Social Cognitive theories, the Health Belief Model, the Theory of Reasoned Action and its successors the Theory of Planned Behaviour and the Reasoned Action Approach, and the Transtheoretical model. We discuss three examples of how behaviour change theories have been applied in computer-based interventions: a system to aid users to quit smoking, a decision aid for choice of breast cancer therapy, and an internet-based exercise program for reducing cardiovascular risk. We also discuss misapplication of theory, and reflect on how these theories can best be used. Behaviour change theory can be applied in health informatics interventions in several ways; for example, to select participants for a particular intervention, to shape the content of the intervention to effectively influence behaviour, or to tailor content to individual needs. Application of these theories to provide personalized advice ("decision support") is a young but promising area of research, and could inform other decision support interventions, including those that provide support for clinicians.

Correction to: How can clinicians, specialty societies and others evaluate and improve the quality of apps for patient use?

Since the publication of this article [1] it has come to my attention that it contains an error in which the y-axis in Fig. 1 was inverted, thus incorrectly displaying a weak negative correlation rather than a weak positive one. This error was introduced as the order of the data on which Fig. 2 was based [2] was misread. The corrected version of Fig. 2 can be seen below, in which a weak positive correlation is now displayed. This does not change the general point, that app users and app stores appear to take little notice of the source of information on which apps are based. I apologise to readers for this error.

DIGITAL TECHNOLOGY: Opportunities and barriers for usage of personal health records in hospital - report from a -workshop of the Health Informatics Unit at the Royal -College of Physicians.

Personal health records (PHRs) are thought to offer benefits and are promoted by health policy makers and some healthcare systems. Evidence for usage by patients in hospital is limited. This article reports a one-day workshop hosted by the Royal College of Physicians that considered the evidence of the value to patients and others, the challenges to adoption and use of PHRs and sought to identify the practical and research questions that need to be answered. The purpose of this article is to provide readers with an overview of the issues and possible future for hospital application of PHRs in the UK's NHS, especially for supporting self-care, family carers and advancing person-centred care. It aims to share the experience and ideas of those taking part in the workshop and reference resources that we have found useful while highlighting areas for future research.

A review of measurement practice in studies of clinical decision support systems 1998-2017.

OBJECTIVE: To assess measurement practice in clinical decision support evaluation studies. MATERIALS AND METHODS: We identified empirical studies evaluating clinical decision support systems published from 1998 to 2017. We reviewed titles, abstracts, and full paper contents for evidence of attention to measurement validity, reliability, or reuse. We used Friedman and Wyatt's typology to categorize the studies. RESULTS: There were 391 studies that met the inclusion criteria. Study types in this cohort were primarily field user effect studies (n = 210) or problem impact studies (n = 150). Of those, 280 studies (72%) had no evidence of attention to measurement methodology, and 111 (28%) had some evidence with 33 (8%) offering validity evidence; 45 (12%) offering reliability evidence; and 61 (16%) reporting measurement artefact reuse. DISCUSSION: Only 5 studies offered validity assessment within the study. Valid measures were predominantly observed in problem impact studies with the majority of measures being clinical or patient reported outcomes with validity measured elsewhere. CONCLUSION: Measurement methodology is frequently ignored in empirical studies of clinical decision support systems and particularly so in field user effect studies. Authors may in fact be attending to measurement considerations and not reporting this or employing methods of unknown validity and reliability in their studies. In the latter case, reported study results may be biased and effect sizes misleading. We argue that replication studies to strengthen the evidence base require greater attention to measurement practice in health informatics research.

Real-world evidence was feasible for estimating effectiveness of chemotherapy in breast cancer: a cohort study.

OBJECTIVE: Evidence-based guidelines recommend adjuvant chemotherapy in early stage breast cancer whenever treatment benefit is considered sufficient to outweigh the associated risks. However, many groups of patients were either excluded from or underrepresented in the clinical trials that form the evidence base for this recommendation. This study aims to determine whether using administrative health care data-real world data-and econometric methods for causal analysis to provide "real world evidence" (RWE) are feasible methods for addressing this gap. METHODS: Cases of primary breast cancer in women from 2001 to 2015 were extracted from the Scottish cancer registry (SMR06) and linked to other routine health records (inpatient and outpatient visits). Four methods were used to estimate the effect of adjuvant chemotherapy on disease-specific and overall mortality: (1) regression with adjustment for covariates, (2) propensity score matching, (3) instrumental variables analysis, and (4) regression discontinuity design. Hazard ratios for breast cancer mortality and all-cause mortality were compared to those from a meta-analysis of randomized trials. RESULTS: A total of 39,805 cases were included in the analyses. Regression adjustment, propensity score matching, and instrumental variables were feasible, whereas regression discontinuity was not. Effectiveness estimates were similar between RWE and randomized trials for breast cancer mortality but not for all-cause mortality. CONCLUSIONS: RWE methods are a feasible means to generate estimates of effectiveness of adjuvant chemotherapy in early stage breast cancer. However, such estimates must be interpreted in the context of the available randomized evidence and the potential biases of the observational methods.