Innovation bundles and platforms - a qualitative analysis of health system responses to the COVID-19 pandemic.

BACKGROUND: Health systems underwent substantial changes to respond to COVID-19. Learning from the successes and failures of health system COVID-19 responses may help us understand how future health service responses can be designed to be both effective and sustainable. This study aims to identify the role that innovation played in crafting health service responses during the COVID-19 pandemic. METHODS: Semi-structured interviews were conducted online, exploring 19 health professionals' experiences in responding to COVID-19 in a large State health system in Australia. The data were collected from April to September 2022 and analysed utilising constant comparative analysis. The degree of innovation in health service responses was assessed by comparing them to pre-pandemic services using 5 categories adopted from the IMPISCO (Investigators, Methods, Population, Intervention, Setting, Comparators and Outcomes) framework, which classifies interventional fidelity as: 1/ Identical: No differences are found between health services; 2/ Substitution with alternatives that perform the same function, 3/ In-class replacement with elements that delivers roughly the same functionality, 4/ Augmentation with new functions, 5/ Creation of new elements. Services were decomposed into bundles and fidelity labels were assigned to individual bundle elements. RESULTS: New services were typically created by reconfiguring existing ones rather than being created de novo. The presence of pre-existing infrastructure (foundational technologies) was seen as critical in mounting fast health service responses. Absence of infrastructure was associated with delays and impaired system responses. CONCLUSIONS: The need to reconfigure rapidly and use infrastructure to support this suggests we reconceive health services as a platform (a general-purpose service upon which other elements can be added for specific functions), where a common core service (such as a primary care practice) can be extended by adding specialised functions using mediators which facilitate the connection (such as virtual service capabilities). Innovation can be costly and time consuming in crises, and during the COVID-19 pandemic, innovations were typically patched together from pre-existing services. The notion of platforms seems a promising way to prepare the health system for future shocks.

Strategies and tactics to reduce the impact of healthcare on climate change: systematic review.

OBJECTIVE: To review the international literature and assess the ways healthcare systems are mitigating and can mitigate their carbon footprint, which is currently estimated to be more than 4.4% of global emissions. DESIGN: Systematic review of empirical studies and grey literature to examine how healthcare services and institutions are limiting their greenhouse gas (GHG) emissions. DATA SOURCES: Eight databases and authoritative reports were searched from inception dates to November 2023. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Teams of investigators screened relevant publications against the inclusion criteria (eg, in English; discussed impact of healthcare systems on climate change), applying four quality appraisal tools, and results are reported in accordance with PRISMA (preferred reporting items for systematic reviews and meta-analyses). RESULTS: Of 33 737 publications identified, 32 998 (97.8%) were excluded after title and abstract screening; 536 (72.5%) of the remaining publications were excluded after full text review. Two additional papers were identified, screened, and included through backward citation tracking. The 205 included studies applied empirical (n=88, 42.9%), review (n=60, 29.3%), narrative descriptive (n=53, 25.9%), and multiple (n=4, 2.0%) methods. More than half of the publications (51.5%) addressed the macro level of the healthcare system. Nine themes were identified using inductive analysis: changing clinical and surgical practices (n=107); enacting policies and governance (n=97); managing physical waste (n=83); changing organisational behaviour (n=76); actions of individuals and groups (eg, advocacy, community involvement; n=74); minimising travel and transportation (n=70); using tools for measuring GHG emissions (n=70); reducing emissions related to infrastructure (n=63); and decarbonising the supply chain (n=48). CONCLUSIONS: Publications presented various strategies and tactics to reduce GHG emissions. These included changing clinical and surgical practices; using policies such as benchmarking and reporting at a facility level, and financial levers to reduce emissions from procurement; reducing physical waste; changing organisational culture through workforce training; supporting education on the benefits of decarbonisation; and involving patients in care planning. Numerous tools and frameworks were presented for measuring GHG emissions, but implementation and evaluation of the sustainability of initiatives were largely missing. At the macro level, decarbonisation approaches focused on energy grid emissions, infrastructure efficiency, and reducing supply chain emissions, including those from agriculture and supply of food products. Decarbonisation mechanisms at the micro and meso system levels ranged from reducing low value care, to choosing lower GHG options (eg, anaesthetic gases, rescue inhalers), to reducing travel. Based on these strategies and tactics, this study provides a framework to support the decarbonisation of healthcare systems. SYSTEMATIC REVIEW REGISTRATION: PROSPERO: CRD42022383719.

A qualitative analysis of health service problems and the strategies used to manage them in the COVID-19 pandemic: exploiting generic and context-specific approaches.

BACKGROUND: The COVID-19 pandemic disrupted health systems around the globe. Lessons from health systems responses to these challenges may help design effective and sustainable health system responses for future challenges. This study aimed to 1/ identify the broad types of health system challenges faced during the pandemic and 2/ develop a typology of health system response to these challenges. METHODS: Semi-structured one-on-one online interviews explored the experience of 19 health professionals during COVID-19 in a large state health system in Australia. Data were analysed using constant comparative analysis utilising a sociotechnical system lens. RESULTS: Participants described four overarching challenges: 1/ System overload, 2/ Barriers to decision-making, 3/ Education or training gaps, and 4/ Limitations of existing services. The limited time often available to respond meant that specific and well-designed strategies were often not possible, and more generic strategies that relied on the workforce to modify solutions and repair unexpected gaps were common. For example, generic responses to system overload included working longer hours, whilst specific strategies utilised pre-existing technical resources (e.g. converting non-emergency wards into COVID-19 wards). CONCLUSION: During the pandemic, it was often not possible to rely on mature strategies to frame responses, and more generic, emergent approaches were commonly required when urgent responses were needed. The degree to which specific strategies were ready-to-hand appeared to dictate how much a strategy relied on such generic approaches. The workforce played a pivotal role in enabling emergent responses that required dealing with uncertainties.

Behavioural 'nudging' interventions to reduce low-value care for low back pain in the emergency department (NUDG-ED): protocol for a 2×2 factorial, before-after, cluster randomised trial.

INTRODUCTION: Opioids and imaging are considered low-value care for most people with low back pain. Yet around one in three people presenting to the emergency department (ED) will receive imaging, and two in three will receive an opioid. NUDG-ED aims to determine the effectiveness of two different behavioural 'nudge' interventions on low-value care for ED patients with low back pain. METHODS AND ANALYSIS: NUDG-ED is a 2×2 factorial, open-label, before-after, cluster randomised controlled trial. The trial includes 8 ED sites in Sydney, Australia. Participants will be ED clinicians who manage back pain, and patients who are 18 years or over presenting to ED with musculoskeletal back pain. EDs will be randomly assigned to receive (i) patient nudges, (ii) clinician nudges, (iii) both interventions or (iv) no nudge control. The primary outcome will be the proportion of encounters in ED for musculoskeletal back pain where a person received a non-indicated lumbar imaging test, an opioid at discharge or both. We will require 2416 encounters over a 9-month study period (3-month before period and 6-month after period) to detect an absolute difference of 10% in use of low-value care due to either nudge, with 80% power, alpha set at 0.05 and assuming an intra-class correlation coefficient of 0.10, and an intraperiod correlation of 0.09. Patient-reported outcome measures will be collected in a subsample of patients (n≥456) 1 week after their initial ED visit. To estimate effects, we will use a multilevel regression model, with a random effect for cluster and patient, a fixed effect indicating the group assignment of each cluster and a fixed effect of time. ETHICS AND DISSEMINATION: This study has ethical approval from Southwestern Sydney Local Health District Human Research Ethics Committee (2023/ETH00472). We will disseminate the results of this trial via media, presenting at conferences and scientific publications. TRIAL REGISTRATION NUMBER: ACTRN12623001000695.

Assessing the Safety of a New Clinical Decision Support System for a National Helpline.

We assessed the safety of a new clinical decision support system (CDSS) for nurses on Australia's national consumer helpline. Accuracy and safety of triage advice was assessed by testing the CDSS using 78 standardised patient vignettes (48 published and 30 proprietary). Testing was undertaken in two cycles using the CDSS vendor's online evaluation tool (Cycle 1: 47 vignettes; Cycle 2: 41 vignettes). Safety equivalence was examined by testing the existing CDSS with the 47 vignettes from Cycle 1. The new CDSS triaged 66% of vignettes correctly compared to 57% by the existing CDSS. 15% of vignettes were overtriaged by the new CDSS compared to 28% by the existing CDSS. 19% of vignettes were undertriaged by the new CDSS compared to 15% by the existing CDSS. Overall performance of the new CDSS appears consistent and comparable with current studies. The new CDSS is at least as safe as the old CDSS.

Automating the Identification of Safety Events Involving Machine Learning-Enabled Medical Devices.

With growing use of machine learning (ML)-enabled medical devices by clinicians and consumers safety events involving these systems are emerging. Current analysis of safety events heavily relies on retrospective review by experts, which is time consuming and cost ineffective. This study develops automated text classifiers and evaluates their potential to identify rare ML safety events from the US FDA's MAUDE. Four stratified classifiers were evaluated using a real-world data distribution with different feature sets: report text; text and device brand name; text and generic device type; and all information combined. We found that stratified classifiers using the generic type of devices were the most effective technique when tested on both stratified (F1-score=85%) and external datasets (precision=100%). All true positives on the external dataset were consistently identified by the three stratified classifiers, indicating the ensemble results from them can be used directly to monitor ML events reported to MAUDE.

The 2-week systematic review (2weekSR) method was successfully blind-replicated by another team: a case study.

OBJECTIVE: To assess the replicability of a 2-week systematic review (index 2weekSR) created with the assistance of automation tools using the fidelity method. METHODS: A Preferred Reporting Items for Systematic reviews and Meta-Analyses compliant SR protocol was developed based on the published information of the index 2weekSR study. The replication team consisted of three reviewers. Two reviewers blocked off time during the replication. The total time to complete tasks and the meta-analysis results were compared with the index 2weekSR study. Review process fidelity scores (FSs) were calculated for review methods and outcomes. Barriers to completing the replication were identified. RESULTS: The review was completed over 63 person-hours (11 workdays/15 calendar days). A FS of 0.95 was achieved for the methods, with 3 (of 8) tasks only partially replicated, and an FS of 0.63 for the outcomes, with 6 (of 7) only partially replicated and one task was not replicated. Nonreplication was mainly caused by missing information in the index 2weekSR study that was not required in standard reporting guidelines. The replication arrived at the same conclusions as the original study. CONCLUSION: A 2weekSR study was replicated by a small team of three reviewers supported by automation tools. Including additional information when reporting SRs should improve their replicability.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings.

AIMS AND OBJECTIVES: To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings. METHOD: PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised. RESULTS: AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making. CONCLUSION: AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.

Clinical decision support versus a paper-based protocol for massive transfusion: Impact on decision outcomes in a simulation study.

BACKGROUND: Management of major hemorrhage frequently requires massive transfusion (MT) support, which should be delivered effectively and efficiently. We have previously developed a clinical decision support system (CDS) for MT using a multicenter multidisciplinary user-centered design study. Here we examine its impact when administering a MT. STUDY DESIGN AND METHODS: We conducted a randomized simulation trial to compare a CDS for MT with a paper-based MT protocol for the management of simulated hemorrhage. A total of 44 specialist physicians, trainees (residents), and nurses were recruited across critical care to participate in two 20-min simulated bleeding scenarios. The primary outcome was the decision velocity (correct decisions per hour) and overall task completion. Secondary outcomes included cognitive workload and System Usability Scale (SUS). RESULTS: There was a statistically significant increase in decision velocity for CDS-based management (mean 8.5 decisions per hour) compared to paper based (mean 6.9 decisions per hour; p .003, 95% CI 0.6-2.6). There was no significant difference in the overall task completion using CDS-based management (mean 13.3) compared to paper-based (mean 13.2; p .92, 95% CI -1.2-1.3). Cognitive workload was statistically significantly lower using the CDS compared to the paper protocol (mean 57.1 vs. mean 64.5, p .005, 95% CI 2.4-12.5). CDS usability was assessed as a SUS score of 82.5 (IQR 75-87.5). DISCUSSION: Compared to paper-based management, CDS-based MT supports more time-efficient decision-making by users with limited CDS training and achieves similar overall task completion while reducing cognitive load. Clinical implementation will determine whether the benefits demonstrated translate to improved patient outcomes.

Post-implementation optimization of medication alerts in hospital computerized provider order entry systems: a scoping review.

OBJECTIVES: A scoping review identified interventions for optimizing hospital medication alerts post-implementation, and characterized the methods used, the populations studied, and any effects of optimization. MATERIALS AND METHODS: A structured search was undertaken in the MEDLINE and Embase databases, from inception to August 2023. Articles providing sufficient information to determine whether an intervention was conducted to optimize alerts were included in the analysis. Snowball analysis was conducted to identify additional studies. RESULTS: Sixteen studies were identified. Most were based in the United States and used a wide range of clinical software. Many studies used inpatient cohorts and conducted more than one intervention during the trial period. Alert types studied included drug-drug interactions, drug dosage alerts, and drug allergy alerts. Six types of interventions were identified: alert inactivation, alert severity reclassification, information provision, use of contextual information, threshold adjustment, and encounter suppression. The majority of interventions decreased alert quantity and enhanced alert acceptance. Alert quantity decreased with alert inactivation by 1%-25.3%, and with alert severity reclassification by 1%-16.5% in 6 of 7 studies. Alert severity reclassification increased alert acceptance by 4.2%-50.2% and was associated with a 100% acceptance rate for high-severity alerts when implemented. Clinical errors reported in 4 studies were seen to remain stable or decrease. DISCUSSION: Post-implementation medication optimization interventions have positive effects for clinicians when applied in a variety of settings. Less well reported are the impacts of these interventions on the clinical care of patients, and how endpoints such as alert quantity contribute to changes in clinician and pharmacist perceptions of alert fatigue. CONCLUSION: Well conducted alert optimization can reduce alert fatigue by reducing overall alert quantity, improving clinical acceptance, and enhancing clinical utility.

The standard problem.

OBJECTIVE: This article proposes a framework to support the scientific research of standards so that they can be better measured, evaluated, and designed. METHODS: Beginning with the notion of common models, the framework describes the general standard problem-the seeming impossibility of creating a singular, persistent, and definitive standard which is not subject to change over time in an open system. RESULTS: The standard problem arises from uncertainty driven by variations in operating context, standard quality, differences in implementation, and drift over time. As a result, fitting work using conformance services is needed to repair these gaps between a standard and what is required for real-world use. To guide standards design and repair, a framework for measuring performance in context is suggested, based on signal detection theory and technomarkers. Based on the type of common model in operation, different conformance strategies are identified: (1) Universal conformance (all agents access the same standard); (2) Mediated conformance (an interoperability layer supports heterogeneous agents); and (3) Localized conformance (autonomous adaptive agents manage their own needs). Conformance methods include incremental design, modular design, adaptors, and creating interactive and adaptive agents. DISCUSSION: Machine learning should have a major role in adaptive fitting. Research to guide the choice and design of conformance services may focus on the stability and homogeneity of shared tasks, and whether common models are shared ahead of time or adjusted at task time. CONCLUSION: This analysis conceptually decouples interoperability and standardization. While standards facilitate interoperability, interoperability is achievable without standardization.

Clinical named entity recognition and relation extraction using natural language processing of medical free text: A systematic review.

BACKGROUND: Natural Language Processing (NLP) applications have developed over the past years in various fields including its application to clinical free text for named entity recognition and relation extraction. However, there has been rapid developments the last few years that there's currently no overview of it. Moreover, it is unclear how these models and tools have been translated into clinical practice. We aim to synthesize and review these developments. METHODS: We reviewed literature from 2010 to date, searching PubMed, Scopus, the Association of Computational Linguistics (ACL), and Association of Computer Machinery (ACM) libraries for studies of NLP systems performing general-purpose (i.e., not disease- or treatment-specific) information extraction and relation extraction tasks in unstructured clinical text (e.g., discharge summaries). RESULTS: We included in the review 94 studies with 30 studies published in the last three years. Machine learning methods were used in 68 studies, rule-based in 5 studies, and both in 22 studies. 63 studies focused on Named Entity Recognition, 13 on Relation Extraction and 18 performed both. The most frequently extracted entities were "problem", "test" and "treatment". 72 studies used public datasets and 22 studies used proprietary datasets alone. Only 14 studies defined clearly a clinical or information task to be addressed by the system and just three studies reported its use outside the experimental setting. Only 7 studies shared a pre-trained model and only 8 an available software tool. DISCUSSION: Machine learning-based methods have dominated the NLP field on information extraction tasks. More recently, Transformer-based language models are taking the lead and showing the strongest performance. However, these developments are mostly based on a few datasets and generic annotations, with very few real-world use cases. This may raise questions about the generalizability of findings, translation into practice and highlights the need for robust clinical evaluation.

Wearable-Triggered Ecological Momentary Assessments Are Feasible in People With Advanced Cancer and Their Family Caregivers: Feasibility Study from an Outpatient Palliative Care Clinic at a Cancer Center.

Background: Emerging digital health approaches could play a role in better personalized palliative care. Aim: We conducted a feasibility study testing wearable sensor (WS)-triggered ecological momentary assessments (EMAs) and electronic patient-reported outcomes in community palliative care with patient-caregiver dyads. Design: All wore consumer-grade WS for five weeks. Sensor-detected "stress" (heart rate variability algorithm) that passed individualized thresholds triggered a short smartphone survey. Daily sleep surveys, weekly symptom surveys (Integrated Palliative care Outcome Scale), and a poststudy experience survey were conducted. Setting/Participants: Fifteen dyads (n = 30) were recruited from an outpatient palliative care clinic for people with cancer. Results: Daytime sensor wear-time had 73% adherence. Participants perceived value in this support. Quantity and severity of "stress" events were higher in patients. Sleep disturbance was similar but for different reasons: patients (physical symptoms) and caregivers (worrying about the patient). Conclusions: EMAs are feasible and valued in community palliative care.

More than algorithms: an analysis of safety events involving ML-enabled medical devices reported to the FDA.

OBJECTIVE: To examine the real-world safety problems involving machine learning (ML)-enabled medical devices. MATERIALS AND METHODS: We analyzed 266 safety events involving approved ML medical devices reported to the US FDA's MAUDE program between 2015 and October 2021. Events were reviewed against an existing framework for safety problems with Health IT to identify whether a reported problem was due to the ML device (device problem) or its use, and key contributors to the problem. Consequences of events were also classified. RESULTS: Events described hazards with potential to harm (66%), actual harm (16%), consequences for healthcare delivery (9%), near misses that would have led to harm if not for intervention (4%), no harm or consequences (3%), and complaints (2%). While most events involved device problems (93%), use problems (7%) were 4 times more likely to harm (relative risk 4.2; 95% CI 2.5-7). Problems with data input to ML devices were the top contributor to events (82%). DISCUSSION: Much of what is known about ML safety comes from case studies and the theoretical limitations of ML. We contribute a systematic analysis of ML safety problems captured as part of the FDA's routine post-market surveillance. Most problems involved devices and concerned the acquisition of data for processing by algorithms. However, problems with the use of devices were more likely to harm. CONCLUSIONS: Safety problems with ML devices involve more than algorithms, highlighting the need for a whole-of-system approach to safe implementation with a special focus on how users interact with devices.

Multicenter, multidisciplinary user-centered design of a clinical decision-support and simulation system for massive transfusion.

BACKGROUND: Managing critical bleeding with massive transfusion (MT) requires a multidisciplinary team, often physically separated, to perform several simultaneous tasks at short notice. This places a significant cognitive load on team members, who must maintain situational awareness in rapidly changing scenarios. Similar resuscitation scenarios have benefited from the use of clinical decision support (CDS) tools. STUDY DESIGN AND METHODS: A multicenter, multidisciplinary, user-centered design (UCD) study was conducted to design a computerized CDS for MT. This study included analysis of the problem context with a cognitive walkthrough, development of a user requirement statement, and co-design with users of prototypes for testing. The final prototype was evaluated using qualitative assessment and the System Usability Scale (SUS). RESULTS: Eighteen participants were recruited across four institutions. The first UCD cycle resulted in the development of four prototype interfaces that addressed the user requirements and context of implementation. Of these, the preferred interface was further developed in the second UCD cycle to create a high-fidelity web-based CDS for MT. This prototype was evaluated by 15 participants using a simulated bleeding scenario and demonstrated an average SUS of 69.3 (above average, SD 16) and a clear interface with easy-to-follow blood product tracking. DISCUSSION: We used a UCD process to explore a highly complex clinical scenario and develop a prototype CDS for MT that incorporates distributive situational awareness, supports multiple user roles, and allows simulated MT training. Evaluation of the impact of this prototype on the efficacy and efficiency of managing MT is currently underway.

Using automated methods to detect safety problems with health information technology: a scoping review.

OBJECTIVE: To summarize the research literature evaluating automated methods for early detection of safety problems with health information technology (HIT). MATERIALS AND METHODS: We searched bibliographic databases including MEDLINE, ACM Digital, Embase, CINAHL Complete, PsycINFO, and Web of Science from January 2010 to June 2021 for studies evaluating the performance of automated methods to detect HIT problems. HIT problems were reviewed using an existing classification for safety concerns. Automated methods were categorized into rule-based, statistical, and machine learning methods, and their performance in detecting HIT problems was assessed. The review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta Analyses extension for Scoping Reviews statement. RESULTS: Of the 45 studies identified, the majority (n = 27, 60%) focused on detecting use errors involving electronic health records and order entry systems. Machine learning (n = 22) and statistical modeling (n = 17) were the most common methods. Unsupervised learning was used to detect use errors in laboratory test results, prescriptions, and patient records while supervised learning was used to detect technical errors arising from hardware or software issues. Statistical modeling was used to detect use errors, unauthorized access, and clinical decision support system malfunctions while rule-based methods primarily focused on use errors. CONCLUSIONS: A wide variety of rule-based, statistical, and machine learning methods have been applied to automate the detection of safety problems with HIT. Many opportunities remain to systematically study their application and effectiveness in real-world settings.

Reliability of crowdsourced data and patient-reported outcome measures in cough-based COVID-19 screening.

Mass community testing is a critical means for monitoring the spread of the COVID-19 pandemic. Polymerase chain reaction (PCR) is the gold standard for detecting the causative coronavirus 2 (SARS-CoV-2) but the test is invasive, test centers may not be readily available, and the wait for laboratory results can take several days. Various machine learning based alternatives to PCR screening for SARS-CoV-2 have been proposed, including cough sound analysis. Cough classification models appear to be a robust means to predict infective status, but collecting reliable PCR confirmed data for their development is challenging and recent work using unverified crowdsourced data is seen as a viable alternative. In this study, we report experiments that assess cough classification models trained (i) using data from PCR-confirmed COVID subjects and (ii) using data of individuals self-reporting their infective status. We compare performance using PCR-confirmed data. Models trained on PCR-confirmed data perform better than those trained on patient-reported data. Models using PCR-confirmed data also exploit more stable predictive features and converge faster. Crowd-sourced cough data is less reliable than PCR-confirmed data for developing predictive models for COVID-19, and raises concerns about the utility of patient reported outcome data in developing other clinical predictive models when better gold-standard data are available.