A distributed approach to the regulation of clinical AI.

Regulation is necessary to ensure the safety, efficacy and equitable impact of clinical artificial intelligence (AI). The number of applications of clinical AI is increasing, which, amplified by the need for adaptations to account for the heterogeneity of local health systems and inevitable data drift, creates a fundamental challenge for regulators. Our opinion is that, at scale, the incumbent model of centralized regulation of clinical AI will not ensure the safety, efficacy, and equity of implemented systems. We propose a hybrid model of regulation, where centralized regulation would only be required for applications of clinical AI where the inference is entirely automated without clinician review, have a high potential to negatively impact the health of patients and for algorithms that are to be applied at national scale by design. This amalgam of centralized and decentralized regulation we refer to as a distributed approach to the regulation of clinical AI and highlight the benefits as well as the pre-requisites and challenges involved.

Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol.

INTRODUCTION: Standards for Reporting of Diagnostic Accuracy Study (STARD) was developed to improve the completeness and transparency of reporting in studies investigating diagnostic test accuracy. However, its current form, STARD 2015 does not address the issues and challenges raised by artificial intelligence (AI)-centred interventions. As such, we propose an AI-specific version of the STARD checklist (STARD-AI), which focuses on the reporting of AI diagnostic test accuracy studies. This paper describes the methods that will be used to develop STARD-AI. METHODS AND ANALYSIS: The development of the STARD-AI checklist can be distilled into six stages. (1) A project organisation phase has been undertaken, during which a Project Team and a Steering Committee were established; (2) An item generation process has been completed following a literature review, a patient and public involvement and engagement exercise and an online scoping survey of international experts; (3) A three-round modified Delphi consensus methodology is underway, which will culminate in a teleconference consensus meeting of experts; (4) Thereafter, the Project Team will draft the initial STARD-AI checklist and the accompanying documents; (5) A piloting phase among expert users will be undertaken to identify items which are either unclear or missing. This process, consisting of surveys and semistructured interviews, will contribute towards the explanation and elaboration document and (6) On finalisation of the manuscripts, the group's efforts turn towards an organised dissemination and implementation strategy to maximise end-user adoption. ETHICS AND DISSEMINATION: Ethical approval has been granted by the Joint Research Compliance Office at Imperial College London (reference number: 19IC5679). A dissemination strategy will be aimed towards five groups of stakeholders: (1) academia, (2) policy, (3) guidelines and regulation, (4) industry and (5) public and non-specific stakeholders. We anticipate that dissemination will take place in Q3 of 2021.

A pragmatic methodology for the evaluation of digital care management in the context of multimorbidity.

Multimorbidity is a defining challenge for health systems and requires coordination of care delivery and care management. Care management is a clinical service designed to remotely engage patients between visits and after discharge in order to support self-management of chronic and emergent conditions, encourage increased use of scheduled care and address the use of unscheduled care. Care management can be provided using digital technology - digital care management. A robust methodology to assess digital care management, or any traditional or digital primary care intervention aimed at longitudinal management of multimorbidity, does not exist outside of randomized controlled trials (RCTs). RCTs are not always generalizable and are also not feasible for most healthcare organizations. We describe here a novel and pragmatic methodology for the evaluation of digital care management that is generalizable to any longitudinal intervention for multimorbidity irrespective of its mode of delivery. This methodology implements propensity matching with bootstrapping to address some of the major challenges in evaluation including identification of robust outcome measures, selection of an appropriate control population, small sample sizes with class imbalances, and limitations of RCTs. We apply this methodology to the evaluation of digital care management at a U.S. payor and demonstrate a 9% reduction in ER utilization, a 17% reduction in inpatient admissions, and a 29% increase in the utilization of preventive medicine services. From these utilization outcomes, we drive forward an estimated cost saving that is specific to a single payor's payment structure for the study time period of $641 per-member-per-month at 3 months. We compare these results to those derived from existing observational approaches, 1:1 and 1:n propensity matching, and discuss the circumstances in which our methodology has advantages over existing techniques. Whilst our methodology focuses on cost and utilization and is applied in the U.S. context, it is applicable to other outcomes such as Patient Reported Outcome Measures (PROMS) or clinical biometrics and can be used in other health system contexts where the challenge of multimorbidity is prevalent.

The myth of generalisability in clinical research and machine learning in health care.

An emphasis on overly broad notions of generalisability as it pertains to applications of machine learning in health care can overlook situations in which machine learning might provide clinical utility. We believe that this narrow focus on generalisability should be replaced with wider considerations for the ultimate goal of building machine learning systems that are useful at the bedside.

"Yes, but will it work for my patients?" Driving clinically relevant research with benchmark datasets.

Benchmark datasets have a powerful normative influence: by determining how the real world is represented in data, they define which problems will first be solved by algorithms built using the datasets and, by extension, who these algorithms will work for. It is desirable for these datasets to serve four functions: (1) enabling the creation of clinically relevant algorithms; (2) facilitating like-for-like comparison of algorithmic performance; (3) ensuring reproducibility of algorithms; (4) asserting a normative influence on the clinical domains and diversity of patients that will potentially benefit from technological advances. Without benchmark datasets that satisfy these functions, it is impossible to address two perennial concerns of clinicians experienced in computational research: "the data scientists just go where the data is rather than where the needs are," and, "yes, but will this work for my patients?" If algorithms are to be developed and applied for the care of patients, then it is prudent for the research community to create benchmark datasets proactively, across specialties. As yet, best practice in this area has not been defined. Broadly speaking, efforts will include design of the dataset; compliance and contracting issues relating to the sharing of sensitive data; enabling access and reuse; and planning for translation of algorithms to the clinical environment. If a deliberate and systematic approach is not followed, not only will the considerable benefits of clinical algorithms fail to be realized, but the potential harms may be regressively incurred across existing gradients of social inequity.

Machine learning health-related applications in low-income and middle-income countries: a scoping review protocol.

INTRODUCTION: Machine learning (ML) has been used in bio-medical research, and recently in clinical and public health research. However, much of the available evidence comes from high-income countries, where different health profiles challenge the application of this research to low/middle-income countries (LMICs). It is largely unknown what ML applications are available for LMICs that can support and advance clinical medicine and public health. We aim to address this gap by conducting a scoping review of health-related ML applications in LMICs. METHODS AND ANALYSIS: This scoping review will follow the methodology proposed by Levac et al. The search strategy is informed by recent systematic reviews of ML health-related applications. We will search Embase, Medline and Global Health (through Ovid), Cochrane and Google Scholar; we will present the date of our searches in the final review. Titles and abstracts will be screened by two reviewers independently; selected reports will be studied by two reviewers independently. Reports will be included if they are primary research where data have been analysed, ML techniques have been used on data from LMICs and they aimed to improve health-related outcomes. We will synthesise the information following evidence mapping recommendations. ETHICS AND DISSEMINATION: The review will provide a comprehensive list of health-related ML applications in LMICs. The results will be disseminated through scientific publications. We also plan to launch a website where ML models can be hosted so that researchers, policymakers and the general public can readily access them.

A Framework for Predicting Impactability of Digital Care Management Using Machine Learning Methods.

Digital care management programs can reduce health care costs and improve quality of care. However, it is unclear how to target patients who are most likely to benefit from these programs ex ante, a shortcoming of current "risk score"-based approaches across many interventions. This study explores a framework to define impactability by using machine learning (ML) models to identify those patients most likely to benefit from a digital health intervention for care management. Anonymized insurance claims data were used from a commercially insured population across several US states and combined with inferred sociodemographic data. The approach involves creating 2 models and the comparative analysis of the methodologies and performances therein. The authors first train a cost prediction model to calculate the differences in predicted (without intervention) versus actual (with onboarding onto digital health platform) health care expenditures for patients (N = 5600). This enables classification impactability if differences in predicted versus actual costs meet a predetermined threshold. Several random forest and logistic regression machine learning models were then trained to accurately categorize new patients as impactable versus not impactable. These parameters are modified through grid search to define the parameters that deliver optimal performance, reaching an overall sensitivity of 0.77 and specificity of 0.65 among all models. This approach shows that impactability for a digital health intervention can be successfully defined using ML methods, thus enabling efficient allocation of resources. This framework is generalizable to analyzing impactability of any intervention and can contribute to realizing closed-loop feedback systems for continuous improvement in health care.

The role of artificial intelligence in orthopaedic surgery.

Despite significant advances in orthopaedic surgery, variability still exists between providers and practice locations, and process inefficiencies are found throughout the health care continuum. Evolving technologies, namely artificial intelligence, challenge the status quo by improving patient care in four areas: diagnosis, management, research and systems analysis. Artificial intelligence shows promise in promoting practice efficiency, personalizing patient care, improving institutional research capacity, and expanding high quality orthopaedic care to lower resource settings. Physicians should be involved in the development of artificial intelligence algorithms to ensure that patients derive maximum benefit from new advances while considering the ethical challenges of implementation.

Using Smartphone Apps to Promote Psychiatric Rehabilitation in a Peer-Led Community Support Program: Pilot Study.

BACKGROUND: Management of severe and persistent mental illness is a complex, resource-intensive challenge for individuals, their families, treaters, and the health care system at large. Community-based rehabilitation, in which peer specialists provide support for individuals managing their own condition, has demonstrated effectiveness but has only been implemented in specialty centers. It remains unclear how the peer-based community rehabilitation model could be expanded, given that it requires significant resources to both establish and maintain. OBJECTIVE: Here, we describe the results from a study of one such program implemented within Waverley Place, a community support program at McLean Hospital, emphasizing psychiatric rehabilitation for individuals with severe and persistent mental illness, as well as describing the challenges encountered during the implementation of the program. Key questions were whether the patients could, and would, successfully use the app. METHODS: The smartphone app offered multiple features relevant to psychiatric rehabilitation, including daily task lists, activity tracking, and text messaging with peer specialists. A 90-day program of activities, goals, and content specific to the community support program was created on the basis of a prior pilot, in collaboration between members of the app development team (WellFrame), and peers, clinical, and research staff associated with the program. Hospital research staff recruited patients into the study, monitored peer and patient engagement, and handled all raw data acquired from the study. RESULTS: Of 100 people approached for the study, a total of 13 provided consent, of which 10 downloaded and used the app. Two patients were unable to complete the app installation. Five used the app regularly as part of their daily lives for at least 20 days of the 90-day program. We were unable to identify any specific factors (eg, clinical or demographic) that affected willingness to consent or engage with the app platform in the very limited sample, although the individuals with significant app use were generally satisfied with the experience. CONCLUSIONS: Smartphone apps may become a useful tool for psychiatric rehabilitation, addressing both psychiatric and co-occurring medical problems. Individualizing functions to each patient and facilitating connection with a certified peer specialist may be an important feature of useful apps. Unlike prior reports emphasizing that patients with schizophrenia will adopt smartphone platforms, we found that implementation of digital tools into existing community support programs for severe and persistent mental illness has many challenges yet to be fully overcome to realize the potential benefits such apps could have to promote systematization and cost reduction for psychiatric rehabilitation.

Technology, seniors, and sense making.

As primary care physicians and leaders of Wellframe, a mobile health company working with payers and physicians groups to extend care between visits for patients with complex comorbidities, Drs Panch and Goodman discuss their experiences building a mobile application used by elderly patients to communicate with clinicians and manage chronic disease.

Are Single-Item Global Ratings Useful for Assessing Health Status?

The research performance of the single-item self-rating In general, would you say your health is: excellent, very good, good, fair, or poor? was evaluated relative to the SF-36 General Health Scale that contains this item, using data for a sample of psychiatric outpatients who had co-occurring chronic physical conditions (N = 177). The scale was more robust than the single-item in cross-sectional validity tests and for predicting 2-year outcomes, but the single-item had stronger discriminant validity as a measure of physical health, especially in post-baseline analyses. Single-item and scale were both sensitive enough to detect change in perceived health over 2 years and a conditional experimental effect on health self-perceptions in a randomized trial. These findings demonstrate that a global single-item can be as valid, reliable, and sensitive as a multi-item scale for longitudinal research purposes, even if the scale performs better in cross-sectional surveys or as a screening measure.

Using Smartphone Apps to Promote Psychiatric and Physical Well-Being.

This pilot study tested the acceptability and usability of a prototype app designed to promote the physical well-being of adults with psychiatric disorders. The application under evaluation, WellWave, promoted walking as a physical exercise, and offered a variety of supportive non-physical activities, including confidential text-messaging with peer staff, and a digital library of readings and videos on recovery from psychiatric illness. Study participants engaged strongly in the app throughout the 4-week study, showing a 94 % mean daily usage rate, and a 73 % mean response rate across all electronic messages and prompts, which approximates the gold standard of 75 % for momentary ecological assessment studies. Seven of the ten study participants averaged two or more walks per week, beginning with 5-min walks and ending with walks lasting 20 min or longer. This responsiveness to the walking prompts, and the overall high rate of engagement in other app features, suggest that adults with psychiatric conditions would welcome and benefit from similar smartphone interventions that promote healthy behaviours in life domains other than exercise. Pilot study results also suggest that smartphone applications can be useful as research tools in the development and testing of theories and practical strategies for encouraging healthy lifestyles. Participants were prompted periodically to rate their own health quality, perceived control over their health, and stage-of-change in adopting a walking routine, and these electronic self-ratings showed acceptable concurrent and discriminant validity, with all participants reporting moderate to high motivation to exercise by the end of the study.

Utility and efficacy of a smartphone application to enhance the learning and behavior goals of traditional cardiac rehabilitation: a feasibility study.

PURPOSE: Most eligible patients do not participate in traditional clinic-based cardiac rehabilitation (CR) despite well-established benefits. Novel approaches to overcome logistic obstacles and increase efficiencies of learning, behavior modification, and exercise surveillance may increase CR participation. In an observational study, the feasibility and utility of a mobile smartphone application for CR, Heart Coach (HC), were assessed as part of standard care. Ultimately, innovative CR models incorporating HC may facilitate better CR usage and value. METHODS: Twenty-six patients enrolled in CR installed HC. Over the next 30 days, they were prompted by HC to complete a daily "task list" that included medications, walking, education (text and videos), and surveys. Cardiac rehabilitation providers monitored each patient's progress through a HC-based Web dashboard and also sent them personalized feedback and support. Completion of the tasks and feedback (qualitative and quantitative) from patients and clinicians were tracked. RESULTS: Patients engaged with HC 90% of days during the study period, with uniformly favorable impact on compliance and adherence. Eighty-three percent of patients reported a positive/very positive HC experience. Providers reported that HC enhanced their provision of therapy by improving communication, clinical insight, patient participation, and program efficiency. CONCLUSIONS: Integrating a mobile care delivery platform into CR was feasible, safe, and agreeable to patients and clinicians. It enhanced patient perceptions of CR care and physician perceptions of the CR caregiving process. Mobile-enabled technologies hold promise to extend the quality and reach of CR, and to better achieve contemporary accountable care goals.