Meaningful Human Control over AI for Health? A Review.

Artificial intelligence is currently changing many areas of society. Especially in health, where critical decisions are made, questions of control must be renegotiated: who is in control when an automated system makes clinically relevant decisions? Increasingly, the concept of meaningful human control (MHC) is being invoked for this purpose. However, it is unclear exactly how this concept is to be understood in health. Through a systematic review, we present the current state of the concept of MHC in health. The results show that there is not yet a robust MHC concept for health. We propose a broader understanding of MHC along three strands of action: enabling, exercising and evaluating control. Taking into account these strands of action and the established rules and processes in the different health sectors, the MHC concept needs to be further developed to avoid falling into two gaps, which we have described as theoretical and labelling gaps.

Evaluation of a clinical decision support system for detection of patients at risk after kidney transplantation.

Patient care after kidney transplantation requires integration of complex information to make informed decisions on risk constellations. Many machine learning models have been developed for detecting patient outcomes in the past years. However, performance metrics alone do not determine practical utility. We present a newly developed clinical decision support system (CDSS) for detection of patients at risk for rejection and death-censored graft failure. The CDSS is based on clinical routine data including 1,516 kidney transplant recipients and more than 100,000 data points. In a reader study we compare the performance of physicians at a nephrology department with and without the CDSS. Internal validation shows AUC-ROC scores of 0.83 for rejection, and 0.95 for graft failure. The reader study shows that predictions by physicians converge toward the CDSS. However, performance does not improve (AUC-ROC; 0.6413 vs. 0.6314 for rejection; 0.8072 vs. 0.7778 for graft failure). Finally, the study shows that the CDSS detects partially different patients at risk compared to physicians. This indicates that the combination of both, medical professionals and a CDSS might help detect more patients at risk for graft failure. However, the question of how to integrate such a system efficiently into clinical practice remains open.

Data justice and data solidarity.

Datafication shapes and gradually transforms societies. Given this impact, issues of justice around data-driven practices have received more and more attention in recent years as shown, for example, by various reports and guidelines on artificial intelligence and data ethics. In this article, we elaborate on and defend two claims. First, these discourses on justice tend to center primarily around conceptions of fairness. We argue that justice in connection with datafication relates to, but ultimately encompasses more than, solely fairness. Second, although it is an important project to clarify what justice in connection with datafication encompasses, we argue that attention toward attitudes and practices of data solidarity have so far been largely overlooked. They are, however, indispensable as a catalytic element to advance toward data justice in practice.

"Nothing works without the doctor:" Physicians' perception of clinical decision-making and artificial intelligence.

Introduction: Artificial intelligence-driven decision support systems (AI-DSS) have the potential to help physicians analyze data and facilitate the search for a correct diagnosis or suitable intervention. The potential of such systems is often emphasized. However, implementation in clinical practice deserves continuous attention. This article aims to shed light on the needs and challenges arising from the use of AI-DSS from physicians' perspectives. Methods: The basis for this study is a qualitative content analysis of expert interviews with experienced nephrologists after testing an AI-DSS in a straightforward usage scenario. Results: The results provide insights on the basics of clinical decision-making, expected challenges when using AI-DSS as well as a reflection on the test run. Discussion: While we can confirm the somewhat expectable demand for better explainability and control, other insights highlight the need to uphold classical strengths of the medical profession when using AI-DSS as well as the importance of broadening the view of AI-related challenges to the clinical environment, especially during treatment. Our results stress the necessity for adjusting AI-DSS to shared decision-making. We conclude that explainability must be context-specific while fostering meaningful interaction with the systems available.

Taking stock of the availability and functions of National Ethics Committees worldwide.

BACKGROUND: National Ethics Committees (NECs) offer important oversight and guidance functions and facilitate public debate on bioethical issues. In an increasingly globalized world where technological advances, multi-national research collaborations, and pandemics are creating ethical dilemmas that transcend national borders, coordination and the joining of efforts among NECs are key. The purpose of this study is to take stock of the current NEC landscape, their varying roles and missions, and the range of bioethical topics on which they deliberated since their inception. METHODS: Data on the availability, functions, and ethical deliberations (publications) of NECs globally were gathered through a systematic search of NEC websites and through contacts known to the authors. The search was conducted in English, French, and Spanish. The data abstraction was done in Excel and included the NEC's country, region, functions, and deliberations on bioethical issues. Deliberation topics were classified into thematic categories through an iterative process of regrouping to arrive at the main set of themes. RESULTS: 124 NECs in 100 countries were identified. 44% of the NECs are in Europe and 47% are in high-income countries. Out of the 1108 retrieved publications, 40% were on bioethics in the context of research, followed by the clinic (28%) and public health issues (22%). The top five topics of these publications were: research ethics (124; 9%), genetics and genomics (62; 6%), organ transplantation (58; 5%), assisted reproductive technology (49; 4%), and end of life (36; 3%). CONCLUSION: Our study makes an important contribution to understanding the current interests and functions of NECs and the range of their bioethics deliberations. By making the data publicly available through this publication, it allows users to conduct tailored analyses and queries based on their interests, and to seek and strengthen collaboration and exchange. It also makes the case for the fruitfulness of developing and maintaining a global repository of current and new deliberations to more effectively advance this field for the greater good of humanity, research, and public health.

A Leap of Faith: Is There a Formula for "Trustworthy" AI?

Trust is one of the big buzzwords in debates about the shaping of society, democracy, and emerging technologies. For example, one prominent idea put forward by the High-Level Expert Group on Artificial Intelligence appointed by the European Commission is that artificial intelligence should be trustworthy. In this essay, we explore the notion of trust and argue that both proponents and critics of trustworthy AI have flawed pictures of the nature of trust. We develop an approach to understanding trust in AI that does not conceive of trust merely as an accelerator for societal acceptance of AI technologies. Instead, we argue, trust is granted through leaps of faith. For this reason, trust remains precarious, fragile, and resistant to promotion through formulaic approaches. We also highlight the significance of distrust in societal deliberation, as it is relevant to trust in various and intricate ways. Among the fruitful aspects of distrust is that it enables individuals to forgo technology if desired, to constrain its power, and to exercise meaningful human control.

Just data? Solidarity and justice in data-driven medicine.

This paper argues that data-driven medicine gives rise to a particular normative challenge. Against the backdrop of a distinction between the good and the right, harnessing personal health data towards the development and refinement of data-driven medicine is to be welcomed from the perspective of the good. Enacting solidarity drives progress in research and clinical practice. At the same time, such acts of sharing could-especially considering current developments in big data and artificial intelligence-compromise the right by leading to injustices and affecting concrete modes of individual self-determination. In order to address this potential tension, two key elements for ethical reflection on data-driven medicine are proposed: the controllability of information flows, including technical infrastructures that are conducive towards controllability, and a paradigm shift towards output-orientation in governance and policy.

Primer on an ethics of AI-based decision support systems in the clinic.

Making good decisions in extremely complex and difficult processes and situations has always been both a key task as well as a challenge in the clinic and has led to a large amount of clinical, legal and ethical routines, protocols and reflections in order to guarantee fair, participatory and up-to-date pathways for clinical decision-making. Nevertheless, the complexity of processes and physical phenomena, time as well as economic constraints and not least further endeavours as well as achievements in medicine and healthcare continuously raise the need to evaluate and to improve clinical decision-making. This article scrutinises if and how clinical decision-making processes are challenged by the rise of so-called artificial intelligence-driven decision support systems (AI-DSS). In a first step, this article analyses how the rise of AI-DSS will affect and transform the modes of interaction between different agents in the clinic. In a second step, we point out how these changing modes of interaction also imply shifts in the conditions of trustworthiness, epistemic challenges regarding transparency, the underlying normative concepts of agency and its embedding into concrete contexts of deployment and, finally, the consequences for (possible) ascriptions of responsibility. Third, we draw first conclusions for further steps regarding a 'meaningful human control' of clinical AI-DSS.

Rapid qualitative review of ethical issues surrounding healthcare for pregnant women or women of reproductive age in epidemic outbreaks.

This article describes, categorizes, and discusses the results of a rapid literature review aiming to provide an overview of the ethical issues and corresponding solutions surrounding pregnancies in epidemic outbreaks. The review was commissioned by the World Health Organization to inform responses to the Zika outbreak that began in 2015. Due to the urgency of the response efforts that needed to be informed by the literature search, a rapid qualitative review of the literature published in PubMed was conducted. The search and analysis were based on the operationalization of 3 key concepts: ethics, pregnancy, and epidemic outbreak. Ethical issues and solutions were interpreted within a principlist framework. The data were analyzed using qualitative content analysis. The search identified 259 publications, of which the full text of 23 papers was read. Of those, 20 papers contained a substantive part devoted to the topic of interest and were therefore analyzed further. We clustered the ethical issues and solutions around 4 themes: uncertainty, harms, autonomy/liberty, and effectiveness. Recognition of the identified ethical issues and corresponding solutions can inform and improve response efforts, public health planning, policies, and decision-making, as well as the activities of medical staff and counselors who practice before, during, or after an epidemic outbreak that affects pregnant women or those of reproductive age. The rapid review format proved to be useful despite its limited data basis and expedited review process.