Evaluating Artificial Intelligence in Clinical Settings-Let Us Not Reinvent the Wheel.

Given the requirement to minimize the risks and maximize the benefits of technology applications in health care provision, there is an urgent need to incorporate theory-informed health IT (HIT) evaluation frameworks into existing and emerging guidelines for the evaluation of artificial intelligence (AI). Such frameworks can help developers, implementers, and strategic decision makers to build on experience and the existing empirical evidence base. We provide a pragmatic conceptual overview of selected concrete examples of how existing theory-informed HIT evaluation frameworks may be used to inform the safe development and implementation of AI in health care settings. The list is not exhaustive and is intended to illustrate applications in line with various stakeholder requirements. Existing HIT evaluation frameworks can help to inform AI-based development and implementation by supporting developers and strategic decision makers in considering relevant technology, user, and organizational dimensions. This can facilitate the design of technologies, their implementation in user and organizational settings, and the sustainability and scalability of technologies.

Automating Electronic Health Record Data Quality Assessment.

Information systems such as Electronic Health Record (EHR) systems are susceptible to data quality (DQ) issues. Given the growing importance of EHR data, there is an increasing demand for strategies and tools to help ensure that available data are fit for use. However, developing reliable data quality assessment (DQA) tools necessary for guiding and evaluating improvement efforts has remained a fundamental challenge. This review examines the state of research on operationalising EHR DQA, mainly automated tooling, and highlights necessary considerations for future implementations. We reviewed 1841 articles from PubMed, Web of Science, and Scopus published between 2011 and 2021. 23 DQA programs deployed in real-world settings to assess EHR data quality (n = 14), and a few experimental prototypes (n = 9), were identified. Many of these programs investigate completeness (n = 15) and value conformance (n = 12) quality dimensions and are backed by knowledge items gathered from domain experts (n = 9), literature reviews and existing DQ measurements (n = 3). A few DQA programs also explore the feasibility of using data-driven techniques to assess EHR data quality automatically. Overall, the automation of EHR DQA is gaining traction, but current efforts are fragmented and not backed by relevant theory. Existing programs also vary in scope, type of data supported, and how measurements are sourced. There is a need to standardise programs for assessing EHR data quality, as current evidence suggests their quality may be unknown.

Neonatal early-onset sepsis calculator safety in an Australian tertiary perinatal centre.

AIM: To determine if any cases of culture-positive neonatal early-onset sepsis (EOS) would be missed using the neonatal EOS calculator, when compared with current guidelines and practices. METHODS: Retrospective audit of all neonates born at ≥35 weeks and admitted to Royal Brisbane and Women's Hospital with EOS from January 2014 to December 2020. A missed case was defined as antibiotic therapy not being recommended within 24 h of birth. Management recommendations according to the neonatal EOS calculator were compared with current guidelines and current practices. RESULTS: There were significantly more missed cases using the neonatal EOS calculator compared to the current guideline and current management groups. Using the neonatal EOS calculator, 11 neonates (35%, 95% confidence interval 19.2-54.6%) would not have received antibiotics by 24 h of age. In comparison, only one neonate (3%, 95% confidence interval 0.1-16.7%) would not have received antibiotics by 24 h of age using the current guidelines. In terms of the current practice in the cohort of patients, two neonates (6%) did not receive antibiotics by 24 h of age. CONCLUSIONS: The significantly higher rate of missed cases using the neonatal EOS calculator compared with current guidelines and practice supports the concerns many neonatologists have regarding safety of the neonatal EOS calculator.

The cytokines interleukin 27 and interferon-γ promote distinct Treg cell populations required to limit infection-induced pathology.

Interferon-γ (IFN-γ) promotes a population of T-bet(+) CXCR3(+) regulatory T (Treg) cells that limit T helper 1 (Th1) cell-mediated pathology. Our studies demonstrate that interleukin-27 (IL-27) also promoted expression of T-bet and CXCR3 in Treg cells. During infection with Toxoplasma gondii, a similar population emerged that limited T cell responses and was dependent on IFN-γ in the periphery but on IL-27 at mucosal sites. Transfer of Treg cells ameliorated the infection-induced pathology observed in Il27(-/-) mice, and this was dependent on their ability to produce IL-10. Microarray analysis revealed that Treg cells exposed to either IFN-γ or IL-27 have distinct transcriptional profiles. Thus, IFN-γ and IL-27 have different roles in Treg cell biology and IL-27 is a key cytokine that promotes the development of Treg cells specialized to control Th1 cell-mediated immunity at local sites of inflammation.

Measuring the operational impact of digitized hospital records: a mixed methods study.

BACKGROUND: Digitized (scanned) medical records have been seen as a means for hospitals to reduce costs and improve access to records. However, clinical usability of digitized records can potentially have negative effects on productivity. METHODS: Data were collected during follow-up outpatient consultations in two NHS hospitals by non-clinical observers using a work sampling approach in which pre-defined categories of clinician time usage were specified. Quantitative data was analysed using two-way ANOVA models and the Mann-Whitney U test. A focus group was held with clinicians to qualitatively explore their experiences using digitized medical records. The quantitative and qualitative results were synthesized. RESULTS: Four hundred six consultations were observed. Using paper records, there was a significant difference in consultation times between hospitals (p = 0.016) and a significant difference in consultation times between specialties within hospitals (p = 0.003). Using digitized records there was a significant difference in consultation times between specialties within a hospital (p = 0.001). Excluding outliers, there was no significant difference between consultation times using digitized records compared with consultations using paper records in the same hospital, either at site (p > =0.285) or specialty level (p > =0.122). With digitized records at site A, two out of three specialties showed a significant increase in time spent searching computer records (p < =0.010, Δ = 01:50-07:10) and one specialty had a corresponding reduction in time spent searching paper records (p = 0.015, Δ = -00:28). Site B showed a notable increase in direct patient care (p < 0.001, Δ = 04:20-06:00) and time spent searching computer records (p < =0.043, Δ = 00:10-01:40) and reductions in the other time categories. The focus group confirmed that the most recent clinical letter was a vital document in the patient record, often containing most of the required information. Concerns were expressed about consistency of scanning practice, causing uncertainty about what could be relied upon to exist in the digitized record. Benefits of digitized records included: access from multiple locations, better prepared ward rounds, improved inpatient handovers and an improved timeline of patient events. Limitations of digitized records included: increased complexity of creating a patient summary, display of specialised content such as hand-drawn diagrams, inability to quickly flick through the pages to find relevant content. CONCLUSIONS: Digitized medical records can be implemented without detrimental operational impact. Inherent differences between specialties can outweigh the differences between paper and digitized records. Clear and consistent operational processes are vital for the reliability and usability of digitized medical records. Divergent views about usability (such as whether patient summary information is better or worse) may reflect familiarity with features of the digitized record.

Diagnosing limb paresis and paralysis in sheep.

Paresis and paralysis are uncommon problems in sheep but are likely to prompt farmers to seek veterinary advice. A thorough and logical approach can aid in determining the cause of the problem and highlighting the benefit of veterinary involvement. While this may not necessarily alter the prognosis for an individual animal, it can help in formulating preventive measures and avoid the costs - both in economic and in welfare terms - of misdirected treatment. Distinguishing between central and peripheral lesions is most important, as the relative prognoses are markedly different, and this can often be achieved with minimal equipment. This article describes an approach to performing a neurological examination of the ovine trunk and limbs, the ancillary tests available and the common and important causes of paresis and paralysis in sheep.

Erratum to: 'Clinical presentation, auscultation recordings, ultrasonographic findings and treatment response of 12 adult cattle with chronic suppurative pneumonia: case study'.

[This corrects the article DOI: 10.1186/2046-0481-66-5.].

Evidence of a high incidence of subclinically affected calves in a herd of cattle with fatal cases of Bovine Neonatal Pancytopenia (BNP).

BACKGROUND: Bovine Neonatal Pancytopenia (BNP) is a disease of calves characterised by bone marrow trilineage hypoplasia, mediated by ingestion of alloantibodies in colostrum. Suspected subclinical forms of BNP have been reported, suggesting that observed clinical cases may not represent the full extent of the disease. However to date there are no objective data available on the incidence of subclinical disease or its temporal distribution. This study aimed to 1) ascertain whether subclinical BNP occurs and, if so, to determine the incidence on an affected farm and 2) determine whether there is evidence of temporal clustering of BNP cases on this farm. To achieve these aims, haematological screening of calves born on the farm during one calving season was carried out, utilising blood samples collected at defined ages. These data were then analysed in comparison to data from both known BNP-free control animals and histopathologically confirmed BNP cases. An ordinal logistic regression model was used to create a composite haematology score to predict the probabilities of calves being normal, based on their haematology measurements at 10-14 days old. RESULTS: This study revealed that 15% (21 of 139) of the clinically normal calves on this farm had profoundly abnormal haematology (<5% chance of being normal) and could be defined as affected by subclinical BNP. Together with clinical BNP cases, this gave the study farm a BNP incidence of 18%. Calves with BNP were found to be distributed throughout the calving period, with no clustering, and no significant differences in the date of birth of cases or subclinical cases were found compared to the rest of the calves. This study did not find any evidence of increased mortality or increased time from birth to sale in subclinical BNP calves but, as the study only involved a single farm and adverse effects may be determined by other inter-current diseases it remains possible that subclinical BNP has a detrimental impact on the health and productivity of calves under certain circumstances. CONCLUSIONS: Subclinical BNP was found to occur at a high incidence in a herd of cattle with fatal cases of BNP.

A national survey of trace organic contaminants in Australian rivers.

Trace organic contaminant (TrOC) studies in Australia have, to date, focused on wastewater effluents, leaving a knowledge gap of their occurrence and risk in freshwater environments. This study measured 42 TrOCs including industrial compounds, pesticides, and pharmaceuticals and personal care products by liquid chromatography tandem mass spectrometry at 73 river sites across Australia quarterly for 1 yr. Trace organic contaminants were found in 92% of samples, with a median of three compounds detected per sample (maximum 18). The five most commonly detected TrOCs were the pharmaceuticals salicylic acid (82%, maximum = 1530 ng/L), paracetamol (also known as acetaminophen; 45%, maximum = 7150 ng/L), and carbamazepine (27%, maximum = 682 ng/L), caffeine (65%, maximum = 3770 ng/L), and the flame retardant (2-chloroethyl) phosphate (44%, maximum = 184 ng/L). Pesticides were detected in 28% of the samples. To determine the risk posed by the detected TrOCs to the aquatic environment, hazard quotients were calculated by dividing the maximum concentration detected for each compound by the predicted no-effect concentrations. Three of the 42 compounds monitored (the pharmaceuticals carbamazepine and sulfamethoxazole and the herbicide simazine) had a hazard quotient >1, suggesting that they may be causing adverse effects at the most polluted sites. A further 10 compounds had hazard quotients >0.1, indicating a potential risk; these included four pharmaceuticals, three personal care products, and three pesticides. Most compounds had hazard quotients significantly <0.1. The number of TrOCs measured in this study was limited and further investigations are required to fully assess the risk posed by complex mixtures of TrOCs on exposed biota.

Clinical presentation, auscultation recordings, ultrasonographic findings and treatment response of 12 adult cattle with chronic suppurative pneumonia: case study.

Auscultation is considered the critical component of the veterinary clinical examination for the diagnosis of bovine respiratory disease but the accuracy with which adventitious sounds reflect underlying lung pathology remains largely unproven. Modern portable ultrasound machines provide the veterinary practitioner with an inexpensive, non-invasive tool with which to examine the pleural surfaces and superficial lung parenchyma. Simultaneous recording of sounds overlying normal lung and defined pathology allows critical assessment of auscultated sounds in the same animal removing confounding factors such as respiratory rate and thickness of the chest wall (body condition). Twelve cows, referred to the University of Edinburgh Veterinary School, were diagnosed with chronic suppurative pneumonia and enrolled into this prospective study to record and monitor lung sounds, ultrasonographic findings, and response to a standardised antibiotic treatment regimen.Most cows (8/12) had a normal rectal temperature on presentation but all cows had received antibiotic therapy at some time in the previous two weeks and six animals were receiving antibiotic treatment upon admission. All cattle were tachypnoeic (>40 breaths per minute) with frequent and productive coughing, halitosis, and a purulent nasal discharge most noticeable when the head was lowered. Ultrasonographic examination of the chest readily identified pathological changes consistent with severe lung pathology subsequently confirmed as chronic suppurative pneumonia in four cows at necropsy; eight cows recovered well after antibiotic treatment and were discharged two to six weeks after admission. It proved difficult to differentiate increased audibility of normal lung sounds due to tachypnoea from wheezes; coarse crackles were not commonly heard. In general, sounds were reduced in volume over consolidated lung relative to normal lung tissue situated dorsally. Rumen contraction sounds were commonly transmitted over areas of lung pathology.Trueperella (formerly Arcanobacterium) pyogenes was isolated from three of four lung tissue samples at necrospy. Treatment with procaine penicillin for 42 consecutive days resulted in marked improvement with return to normal appetite and improvement in body condition in 8 of 12 cows (67%) where lesions did not extend more than 10-15 cm above the level of the olecranon on both sides of the chest.

Digital health and care: emerging from pandemic times.

In 2020, we published an editorial about the massive disruption of health and care services caused by the COVID-19 pandemic and the rapid changes in digital service delivery, artificial intelligence and data sharing that were taking place at the time. Now, 3 years later, we describe how these developments have progressed since, reflect on lessons learnt and consider key challenges and opportunities ahead by reviewing significant developments reported in the literature. As before, the three key areas we consider are digital transformation of services, realising the potential of artificial intelligence and wise data sharing to facilitate learning health systems. We conclude that the field of digital health has rapidly matured during the pandemic, but there are still major sociotechnical, evaluation and trust challenges in the development and deployment of new digital services.

One Health in a Digital World: Technology, Data, Information and Knowledge.

OBJECTIVES: To describe the origins and growth of the One Health concept and its recent application in One Digital Health. METHODS: Bibliometric review and critical discussion of emergent themes derived from co-occurrence of MeSH keywords. RESULTS: The fundamental interrelationship between human health, animal health and the wider environment has been recognized since ancient times. One Health as a distinct term originated in 2004 and has been a rapidly growing concept of interest in the biomedical literature since 2017. One Digital Health has quickly established itself as a unifying construct that highlights the critical role of technology, data, information and knowledge to facilitate the interdisciplinary collaboration that One Health requires. The principal application domains of One Digital Health to date are in FAIR data integration and analysis, disease surveillance, antimicrobial stewardship and environmental monitoring. CONCLUSIONS: One Health and One Digital Health offer powerful lenses to examine and address crises in our living world. We propose thinking in terms of Learning One Health Systems that can dynamically capture, integrate, analyse and monitor application of data across the biosphere.

Barriers, Priorities and Lessons Learned in Achieving Electronic Health Records Interoperability in Low- and Middle-Income Countries: Workshop Findings.

Electronic health records (EHR) interoperability is a complex topic that continues to gain traction in the digital health landscape. We facilitated a qualitative workshop consisting of domain experts in EHR implementation and health IT managers. The workshop aimed to identify critical barriers to achieving interoperability, priorities for new EHR implementations and lessons learned from managing existing implementations. The workshop highlighted that data modelling and interoperability standards are vital priorities for maternal and child health data services in low- and middle-income countries (LMICs).

Which computable biomedical knowledge objects will be regulated? Results of a UK workshop discussing the regulation of knowledge libraries and software as a medical device.

Introduction: To understand when knowledge objects in a computable biomedical knowledge library are likely to be subject to regulation as a medical device in the United Kingdom. Methods: A briefing paper was circulated to a multi-disciplinary group of 25 including regulators, lawyers and others with insights into device regulation. A 1-day workshop was convened to discuss questions relating to our aim. A discussion paper was drafted by lead authors and circulated to other authors for their comments and contributions. Results: This article reports on those deliberations and describes how UK device regulators are likely to treat the different kinds of knowledge objects that may be stored in computable biomedical knowledge libraries. While our focus is the likely approach of UK regulators, our analogies and analysis will also be relevant to the approaches taken by regulators elsewhere. We include a table examining the implications for each of the four knowledge levels described by Boxwala in 2011 and propose an additional level. Conclusions: If a knowledge object is described as directly executable for a medical purpose to provide decision support, it will generally be in scope of UK regulation as "software as a medical device." However, if the knowledge object consists of an algorithm, a ruleset, pseudocode or some other representation that is not directly executable and whose developers make no claim that it can be used for a medical purpose, it is not likely to be subject to regulation. We expect similar reasoning to be applied by regulators in other countries.

One Health: Insights from Organizational & Social, Technology Assessment and Human Factors Perspectives.

OBJECTIVES: To offer diverse but complementary perspectives on how biomedical and health informatics can be informed by and help to achieve the vision of One Health. METHODS: Overview of key considerations and critical discussion of common themes, barriers and opportunities, based on collaborative review by International Medical Informatics Association (IMIA) working group members active in related fields. RESULTS: Health and care systems are complex sociotechnical systems that need explicit design and implementation strategies to align with the goals of One Health. The evidence-based health informatics paradigm and associated frameworks for evaluation of digital health technologies need to broaden their scope to take full account of the One Health approach. Informatics has specific contributions to make to One Health, for example by improved user experience reducing energy consumption and effective app design enhancing medication adherence. CONCLUSIONS: One Health is inherently intertwined with ergonomic, sociotechnical and evaluation perspectives in biomedical and health informatics. Health is a planetary issue that requires interdisciplinary collaborative action. The theories and principles of biomedical and health informatics offer many opportunities to transform digital health technology to better serve the One Health agenda.

Modelling clinical narrative as computable knowledge: The NICE computable implementation guidance project.

Introduction: Translating narrative clinical guidelines to computable knowledge is a long-standing challenge that has seen a diverse range of approaches. The UK National Institute for Health and Care Excellence (NICE) Content Advisory Board (CAB) aims ultimately to (1) guide clinical decision support and other software developers to increase traceability, fidelity and consistency in supporting clinical use of NICE recommendations, (2) guide local practice audit and intervention to reduce unwarranted variation, (3) provide feedback to NICE on how future recommendations should be developed. Objectives: The first phase of work was to explore a range of technical approaches to transition NICE toward the production of natively digital content. Methods: Following an initial 'collaborathon' in November 2022, the NICE Computable Implementation Guidance project (NCIG) was established. We held a series of workstream calls approximately fortnightly, focusing on (1) user stories and trigger events, (2) information model and definitions, (3) horizon-scanning and output format. A second collaborathon was held in March 2023 to consolidate progress across the workstreams and agree residual actions to complete. Results: While we initially focussed on technical implementation standards, we decided that an intermediate logical model was a more achievable first step in the journey from narrative to fully computable representation. NCIG adopted the WHO Digital Adaptation Kit (DAK) as a technology-agnostic method to model user scenarios, personae, processes and workflow, core data elements and decision-support logic. Further work will address indicators, such as prescribing compliance, and implementation in document templates for primary care patient record systems. Conclusions: The project has shown that the WHO DAK, with some modification, is a promising approach to build technology-neutral logical specifications of NICE recommendations. Implementation of concurrent computable modelling by multidisciplinary teams during guideline development poses methodological and cultural questions that are complex but tractable given suitable will and leadership.

The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision.

Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology. We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes. Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.

Evaluating the potential of effluents and wood feedstocks from pulp and paper mills in Brazil, Canada, and New Zealand to affect fish reproduction: chemical profiling and in vitro assessments.

This study investigates factors affecting reproduction in fish exposed to pulp and paper mill effluents by comparing effluents from countries with varying levels of documented effects. To explore the hypothesis of wood as a common source of endocrine disrupting compounds, feedstocks from each country were analyzed. Analyses included in vitro assays for androgenic activity (binding to goldfish testis androgen receptors), estrogenic activity (yeast estrogen screen), and neurotransmitter enzyme inhibition (monoamine oxidase and glutamic acid decarboxylase). Chemical analyses included conventional extractives, known androgens, and gas chromatograph index (GCI) profiles. All effluents and wood contained androgenic activity, particularly in nonpolar fractions, although known androgens were undetected. Effluents with low suspended solids, having undergone conventional biotreatment had lower androgenic activities. Estrogenic activity was only associated with Brazilian effluents and undetected in wood. All effluents and wood inhibited neurotransmitter enzymes, predominantly in polar fractions. Kraft elemental chlorine free mills were associated with the greatest neurotransmitter inhibition. Effluent and wood GCI profiles were correlated with androgenic activity and neurotransmitter enzyme inhibition. Differences in feedstock bioactivities were not reflected in effluents, implying mill factors mitigate bioactive wood components. No differences in bioactivities could be discerned on the basis of country of origin, thus we predict effluents in regions lacking monitoring would affect fish reproduction and therefore recommend implementing such programs.

Semantic mapping to simplify deployment of HL7 v3 Clinical Document Architecture.

HL7 Version 3 was developed to provide semantically reliable healthcare interoperability but has been perceived as overly complex to implement. This paper describes how semantic mapping can simplify usage of HL7 Version 3 Clinical Document Architecture (CDA) and reports on the rationale for adopting this approach in three particular use cases and lessons learned from the practical application experience. We envision a broad application of the semantic mapping approach in clinical documents, messaging and application design modelling.

Experiences of Creating Computable Knowledge Tutorials Using HL7 Clinical Quality Language.

Computable knowledge artefact development is challenging and often culminates in the development of unique single usage solutions. Libraries of computable knowledge artefacts have the possibility to enhance the Learning Health System in order to improve the benefits of innovation and the decision making of clinicians. This paper aims to discuss the process of creating the use cases and the tutorial material that would enable students to both understand how the interaction between the dataset and the outcome occurs as well as how HL7 Clinical Quality Language can be used to create artefacts of re-usable code.