A riddle, wrapped in a mystery, inside an enigma: How semantic black boxes and opaque artificial intelligence confuse medical decision-making.

The use of artificial intelligence (AI) in healthcare comes with opportunities but also numerous challenges. A specific challenge that remains underexplored is the lack of clear and distinct definitions of the concepts used in and/or produced by these algorithms, and how their real world meaning is translated into machine language and vice versa, how their output is understood by the end user. This "semantic" black box adds to the "mathematical" black box present in many AI systems in which the underlying "reasoning" process is often opaque. In this way, whereas it is often claimed that the use of AI in medical applications will deliver "objective" information, the true relevance or meaning to the end-user is frequently obscured. This is highly problematic as AI devices are used not only for diagnostic and decision support by healthcare professionals, but also can be used to deliver information to patients, for example to create visual aids for use in shared decision-making. This paper provides an examination of the range and extent of this problem and its implications, on the basis of cases from the field of intensive care nephrology. We explore how the problematic terminology used in human communication about the detection, diagnosis, treatment, and prognosis of concepts of intensive care nephrology becomes a much more complicated affair when deployed in the form of algorithmic automation, with implications extending throughout clinical care, affecting norms and practices long considered fundamental to good clinical care.

Interrupting pathways to health inequities in citizen science health research.

Participatory research in multiple forms is rapidly gaining ground and stands to provide valuable benefits in the area of health research. Citizen science, an increasingly popular form of participatory research, aims to involve laypersons in knowledge production and, as such, can offer new insights that may not be readily accessible by the traditional scientific research enterprise. Heralded as an advance in the democratization of science, citizen science stands as a potential player in evidence generation. Yet, while citizen science offers advantages through its emphasis on lay participation in uncovering information likely to be pertinent to improving health outcomes, without deliberate attention to pathways to inequity and oversight, some aspects can operate to perpetuate or exacerbate health inequities. Drawing on health disparities literature to assist in identifying possible pathways, this work aims to provide support for greater attention to 'process', which would include active engagement with intersecting inequities as a way of interrupting pathways to health inequity. If citizen science is to be true to its moniker, it cannot be 'equity-neutral' in a society where structural or systemic inequities are present. It must be equity-promoting. In the health context, the concerns of justice evolve into concerns of differential health impacts. To accomplish this, equity and justice, as well as representation and fair process, become critically important. Accordingly, this paper foregrounds proactive commitment to equity by confronting the reality of social inequalities present in a given society and adapting the project accordingly to minimize downstream health inequities.

Participatory research in multiple forms is rapidly gaining ground and stands to provide valuable benefits in the area of health research. Citizen science, a form of participatory research, aims to involve laypersons in contributing to scientific knowledge. This can offer new insights that may be more difficult for the traditional scientific research institutions. Many see citizen science as an advance in the democratization of science. That is, it is a form of 'science by the people' rather than the exclusive domain of highly trained researchers. Research conducted by citizen scientists may uncover information that can help improve health outcomes, but without adequate attention to process and oversight, some aspects can operate to perpetuate or compound health inequities. This work aims to provide support for engaging with background inequities within a society and using that awareness to inform the planning, design and execution of citizen science projects. To avoid unjust outcomes in which people receive different advantages, health research needs to be conducted in an 'equity-promoting' manner. Citizen science must elevate the importance of 'process' and take a proactive and deliberate approach to investigating social inequalities present in society that may intersect with the project, and adapt the project accordingly.

Using Social Media as a Research Recruitment Tool: Ethical Issues and Recommendations.

The use of social media as a recruitment tool for research with humans is increasing, and likely to continue to grow. Despite this, to date there has been no specific regulatory guidance and there has been little in the bioethics literature to guide investigators and institutional review boards (IRBs) faced with navigating the ethical issues such use raises. We begin to fill this gap by first defending a nonexceptionalist methodology for assessing social media recruitment; second, examining respect for privacy and investigator transparency as key norms governing social media recruitment; and, finally, analyzing three relatively novel aspects of social media recruitment: (i) the ethical significance of compliance with website "terms of use"; (ii) the ethics of recruiting from the online networks of research participants; and (iii) the ethical implications of online communication from and between participants. Two checklists aimed at guiding investigators and IRBs through the ethical issues are included as appendices.

Diversity and inclusion: A hidden additional benefit of Open Data.

The recent imperative by the National Institutes of Health to share scientific data publicly underscores a significant shift in academic research. Effective as of January 2023, it emphasizes that transparency in data collection and dedicated efforts towards data sharing are prerequisites for translational research, from the lab to the bedside. Given the role of data access in mitigating potential bias in clinical models, we hypothesize that researchers who leverage open-access datasets rather than privately-owned ones are more diverse. In this brief report, we proposed to test this hypothesis in the transdisciplinary and expanding field of artificial intelligence (AI) for critical care. Specifically, we compared the diversity among authors of publications leveraging open datasets, such as the commonly used MIMIC and eICU databases, with that among authors of publications relying exclusively on private datasets, unavailable to other research investigators (e.g., electronic health records from ICU patients accessible only to Mayo Clinic analysts). To measure the extent of author diversity, we characterized gender balance as well as the presence of researchers from low- and middle-income countries (LMIC) and minority-serving institutions (MSI) located in the United States (US). Our comparative analysis revealed a greater contribution of authors from LMICs and MSIs among researchers leveraging open critical care datasets (treatment group) than among those relying exclusively on private data resources (control group). The participation of women was similar between the two groups, albeit slightly larger in the former. Notably, although over 70% of all articles included at least one author inferred to be a woman, less than 25% had a woman as a first or last author. Importantly, we found that the proportion of authors from LMICs was substantially higher in the treatment than in the control group (10.1% vs. 6.2%, p<0.001), including as first and last authors. Moreover, we found that the proportion of US-based authors affiliated with a MSI was 1.5 times higher among articles in the treatment than in the control group, suggesting that open data resources attract a larger pool of participants from minority groups (8.6% vs. 5.6%, p<0.001). Thus, our study highlights the valuable contribution of the Open Data strategy to underrepresented groups, while also quantifying persisting gender gaps in academic and clinical research at the intersection of computer science and healthcare. In doing so, we hope our work points to the importance of extending open data practices in deliberate and systematic ways.

Peer review of GPT-4 technical report and systems card.

The study provides a comprehensive review of OpenAI's Generative Pre-trained Transformer 4 (GPT-4) technical report, with an emphasis on applications in high-risk settings like healthcare. A diverse team, including experts in artificial intelligence (AI), natural language processing, public health, law, policy, social science, healthcare research, and bioethics, analyzed the report against established peer review guidelines. The GPT-4 report shows a significant commitment to transparent AI research, particularly in creating a systems card for risk assessment and mitigation. However, it reveals limitations such as restricted access to training data, inadequate confidence and uncertainty estimations, and concerns over privacy and intellectual property rights. Key strengths identified include the considerable time and economic investment in transparent AI research and the creation of a comprehensive systems card. On the other hand, the lack of clarity in training processes and data raises concerns about encoded biases and interests in GPT-4. The report also lacks confidence and uncertainty estimations, crucial in high-risk areas like healthcare, and fails to address potential privacy and intellectual property issues. Furthermore, this study emphasizes the need for diverse, global involvement in developing and evaluating large language models (LLMs) to ensure broad societal benefits and mitigate risks. The paper presents recommendations such as improving data transparency, developing accountability frameworks, establishing confidence standards for LLM outputs in high-risk settings, and enhancing industry research review processes. It concludes that while GPT-4's report is a step towards open discussions on LLMs, more extensive interdisciplinary reviews are essential for addressing bias, harm, and risk concerns, especially in high-risk domains. The review aims to expand the understanding of LLMs in general and highlights the need for new reflection forms on how LLMs are reviewed, the data required for effective evaluation, and addressing critical issues like bias and risk.

From quality improvement to equality improvement projects: A scoping review and framework.

Increasing awareness of health disparities has led to proposals for a pay-for-equity scheme. Implementing such proposals requires systematic methods of collecting and reporting health outcomes for targeted demographics over time. This lays the foundation for a shift from quality improvement projects (QIPs) to equality improvement projects (EQIPs) that could evaluate adherence to standards and progress toward health equity. We performed a scoping review on EQIPs to inform a new framework for quality improvement through a health equity lens. Forty studies implemented an intervention after identifying a disparity compared to 149 others which merely identified group differences. Most evaluated race-based differences and were conducted at the institutional level, with representation in both the inpatient and outpatient settings. EQIPs that improved equity leveraged multidisciplinary expertise, healthcare staff education, and developed tools to track health outcomes continuously. EQIPs can help bridge the inequality gap and form part of an incentivized systematic equality improvement framework.

Quantifying digital health inequality across a national healthcare system.

OBJECTIVES: Digital health inequality, observed as differential utilisation of digital tools between population groups, has not previously been quantified in the National Health Service (NHS). Deployment of universal digital health interventions, including a national smartphone app and online primary care services, allows measurement of digital inequality across a nation. We aimed to measure population factors associated with digital utilisation across 6356 primary care providers serving the population of England. METHODS: We used multivariable regression to test association of population and provider characteristics (including patient demographics, socioeconomic deprivation, disease burden, prescribing burden, geography and healthcare provider resource) with activation of two independent digital services during 2021/2022. RESULTS: We find a significant adjusted association between increased population deprivation and reduced digital utilisation across both interventions. Multivariable regression coefficients for most deprived quintiles correspond to 4.27 million patients across England where deprivation is associated with non-activation of the NHS App. CONCLUSION: Results are concerning for technologically driven widening of healthcare inequalities. Targeted incentive to digital is necessary to prevent digital disparity from becoming health outcomes disparity.

Equity should be fundamental to the emergence of innovation.

The ability of artificial intelligence to perpetuate bias at scale is increasingly recognized. Recently, proposals for implementing regulation that safeguards such discrimination have come under pressure due to the potential of such restrictions stifling innovation within the field. In this formal comment, we highlight the potential dangers of such views and explore key examples that define this relationship between health equity and innovation. We propose that health equity is a vital component of healthcare and should not be compromised to expedite the advancement of results for the few at the expense of vulnerable populations. A data-centered future that works for all will require funding bodies to incentivize equity-focused AI, and organizations must be held accountable for the differential impact of such algorithms post-deployment.

Whose ethics of knowledge? Taking the next step in evaluating knowledge in synthetic biology: a response to Douglas and Savulescu.

The recent proposal by Douglas and Savulescu for an ethics of knowledge provokes a renewed consideration of an enduring issue. Yet, the concept raises significant challenges for procedural and substantive justice. Indeed, the operationalisation of 'an ethics of knowledge' could be as alarming as what it seeks to prevent. While we can acknowledge that there is, and surely always will be, potential for misuse of beneficial science and technology, a contemplated conception of what we ought to not know, devise or disseminate sets before us an enormously complex task. This essay challenges an ethics of knowledge to respond to concerns of procedural and substantive justice. While the concept has a certain appeal, it does not appear to adequately address certain fundamental issues as it is currently presented. Here, the author invites consideration of two primary points: (1) who should decide, based on whose interests? and (2) could such an exercise actually be effective in achieving its goal?

Explainability in medicine in an era of AI-based clinical decision support systems.

The combination of "Big Data" and Artificial Intelligence (AI) is frequently promoted as having the potential to deliver valuable health benefits when applied to medical decision-making. However, the responsible adoption of AI-based clinical decision support systems faces several challenges at both the individual and societal level. One of the features that has given rise to particular concern is the issue of explainability, since, if the way an algorithm arrived at a particular output is not known (or knowable) to a physician, this may lead to multiple challenges, including an inability to evaluate the merits of the output. This "opacity" problem has led to questions about whether physicians are justified in relying on the algorithmic output, with some scholars insisting on the centrality of explainability, while others see no reason to require of AI that which is not required of physicians. We consider that there is merit in both views but find that greater nuance is necessary in order to elucidate the underlying function of explainability in clinical practice and, therefore, its relevance in the context of AI for clinical use. In this paper, we explore explainability by examining what it requires in clinical medicine and draw a distinction between the function of explainability for the current patient versus the future patient. This distinction has implications for what explainability requires in the short and long term. We highlight the role of transparency in explainability, and identify semantic transparency as fundamental to the issue of explainability itself. We argue that, in day-to-day clinical practice, accuracy is sufficient as an "epistemic warrant" for clinical decision-making, and that the most compelling reason for requiring explainability in the sense of scientific or causal explanation is the potential for improving future care by building a more robust model of the world. We identify the goal of clinical decision-making as being to deliver the best possible outcome as often as possible, and find-that accuracy is sufficient justification for intervention for today's patient, as long as efforts to uncover scientific explanations continue to improve healthcare for future patients.

Reducing Anxiety and Stress among Youth in a CBT-Based Equine-Assisted Adaptive Riding Program.

Reining in Anxiety (RiA) is a therapeutic program for youth with mild to moderate anxiety delivered in a therapeutic riding setting by Certified Therapeutic Riding Instructors. RiA was developed after a review of the evidence base for youth anxiety, is manualized, and includes five core CBT components: in vivo exposure, cognitive restructuring, youth psychoeducation, relaxation, and caregiver psychoeducation about anxiety. This study extended findings from a prior RCT that examined (1) the feasibility of collecting saliva samples from horses and children to measure stress (cortisol) and relaxation (oxytocin); (2) whether changes in stress and relaxation occurred both during each lesson and over the course of the 10-week intervention for horses and youth; (3) whether changes in anxiety symptoms, emotional regulation, and self-efficacy found in the first trial were comparable; and (4) if fidelity to the program was reliable. Youth participants (n = 39) ages 6-17 with caregiver-identified mild-to-moderate anxiety participated in a ten-week therapeutic intervention (RiA), which combined adaptive riding and components of CBT. Physiological data and self-report measures were taken at weeks one, four, seven, and ten for the youth and horses. Saliva assays assessed cortisol as a physiological marker of stress and anxiety, and oxytocin as a measure of relaxation. Fidelity data were recorded per session. Anxiety, as measured by caregiver self-reporting, significantly decreased from pre- to post-test, while emotional regulation scores increased. No significant changes in self-efficacy from pre- to post-test were observed. Saliva samples obtained from participants before and after riding sessions showed a consistent decrease in cortisol and a significant increase in oxytocin at two of the four timepoints (Week 1 and Week 7), but no overall pre- to post-test changes. Horse saliva data were collected using a modified bit; there were no significant changes in oxytocin or cortisol, suggesting that the horses did not have an increase in stress from the intervention. RiA may be a promising approach for reducing anxiety and stress among youth, as measured both by self-reported and by physiological measures. Collection of salivary assays for both youth and horses is feasible, and the intervention does not increase stress in the horses. Importantly, RiA can be delivered by adaptive/therapeutic horseback riding instructors in naturalistic (e.g., non-clinic-based) settings. As youth anxiety is a growing public health problem, novel interventions, such as RiA, that can be delivered naturalistically may have the potential to reach more youth and thus improve their quality of life. Further research is needed to examine the comparative value of RiA with other animal-assisted interventions and to assess its cost-effectiveness.

Revisiting caregiver satisfaction with children's mental health services in the United States.

Nearly four decades ago, Unclaimed Children documented the gaps in the United States between mental health programs and caregivers' perspectives about those services for their children. This absence of attention to parent or caregiver perspectives, including their satisfaction with these services, was a key finding of the report, which detailed system failure in caring for youth with mental health needs. Since then, the focus on caregiver satisfaction with children's mental health services has been largely overlooked in research, and when examined has been mostly included as an indicator of the feasibility of program implementation. In striking contrast, overall healthcare system reforms have highlighted the importance of improving consumer's direct experience of care. However, caregiver satisfaction remains largely disconnected to these overall health system reforms, even as reforms focus increasingly on value-based, coordinated and integrated care. In this paper, we review literature from 2010 to 2020, revisit the measurement of caregiver satisfaction, identify how and when it is being measured, and delineate a research agenda to both realign it with health system improvements, refine its focus on expectancies and appropriateness, and root it more firmly in the principles of user experience (UX) and human-centered design (HCD).

Preventing Risk and Promoting Young Children's Mental, Emotional, and Behavioral Health in State Mental Health Systems.

Early neural development and maternal health have critical long-term effects on children's mental health and outcomes later in life. As child mental disorders continue to rise nationwide, a number of states are considering new ways of investing in the critical early childhood period to prevent later poor outcomes and reduce the burden on the mental health system. Because most state mental health authorities (SMHAs) have no dedicated mental health dollars to devote to this early, crucial period of child development, building coalitions is key to implementing prevention and promotion programming. The authors describe two issues-coalition building and contractual considerations-that should be considered as SMHAs develop these types of policies or plan new prevention and promotion initiatives. Coalition building includes establishing the structural conditions for implementing a prevention or promotion initiative, resolving workforce issues (i.e., who will carry the program out), and engaging communities and families in the effort. Contractual considerations include establishing agreed-upon measures and metrics to monitor outcomes, assigning accountability for those outcomes, and delineating realistic time frames for these investments before expecting improved outcomes. The promise of moving services upstream to support early childhood development, to prevent mental health issues from derailing children's development, and to promote children's well-being are goals that are within reach.

Governing AI-Driven Health Research: Are IRBs Up to the Task?

Many are calling for concrete mechanisms of oversight for health research involving artificial intelligence (AI). In response, institutional review boards (IRBs) are being turned to as a familiar model of governance. Here, we examine the IRB model as a form of ethics oversight for health research that uses AI. We consider the model's origins, analyze the challenges IRBs are facing in the contexts of both industry and academia, and offer concrete recommendations for how these committees might be adapted in order to provide an effective mechanism of oversight for health-related AI research.

Evaluation of a Modified Bit Device to Obtain Saliva Samples from Horses.

(1) Background: Accounting for the well-being of equine partners is a responsibility of those engaged in Equine-Assisted Services (EAS). Researchers took heed of this call to action by developing an innovative way to collect data to assess the physiological indicators of stress in equine participants. The collection of saliva is considered to be a minimally invasive method of data collection and is typically performed using a cotton swab; however, in equines, the introduction of a foreign object may induce stress; (2) Methods: Researchers used a modified bit to collect pooled saliva in an effort to further reduce stress during the saliva collection process. Additionally, the collection of pooled saliva, via the bit, increases the opportunity to consider additional analyses, such as oxytocin, which is more reliable in pooled saliva than site-specific saliva captured with a swab; (3) Results: A data analysis demonstrated that ample saliva was captured using the modified bit. Observational data supported that the horses demonstrated fewer physical stress signals to the bit than to the swab. Thus, the modified bit is a feasible and valid method for equine salivary sample collection; (4) Conclusions: The results suggest that the modified bit provides a viable method to collect equine saliva and supports national calls to prioritize animal welfare analysis, specifically for horses used within EAS. Future research should enhance methodological rigor, including in the process and timing, thereby contributing to the bit's validation.

Trends Over a Decade in NIH Funding for Autism Spectrum Disorder Services Research.

Investments in autism spectrum disorder (ASD) research, guided by the Interagency Autism Coordinating Committee (IACC), have focused disproportionately on etiology over a well-established stakeholder priority area: research to improve accessibility and quality of community-based services. This study analyzed National Institutes of Health ASD services research funding from 2008 to 2018 to examine funding patterns, evaluate the impact of IACC objectives, and identify future directions. Approximately 9% of total funds were allocated to services research. This investment remained relatively stable across time and lacked diversity across domains (e.g., area of focus, ages sampled, implementation strategies used). While advancements were observed, including increased prevalence of projects focused on adult samples and on dissemination/implementation and prevention areas, greater investment in service research is critically needed.

Complex calculations: ethical issues in involving at-risk healthy individuals in dementia research.

In dementia research evidence is mounting that therapeutic strategies that target moderate and even mild Alzheimer's disease may be missing the 'therapeutic window'. Given that the neuropathology that leads to Alzheimer's disease probably begins somewhere between 10 and 15 years before symptoms manifest, many believe that the optimal therapeutic strategy would target persons in the earliest phases of disease development or even earlier. This would include, for example, persons with prodromal Alzheimer's and even persons who are deemed at risk. Given the nature of research involving the central nervous system, it is conceivable that some therapeutic investigations may involve an increase over minimal risk. This paper examines how, in dementia research, at-risk persons, although healthy, bring multiple and intersecting vulnerabilities to the prospect of research participation even though they are clinically healthy. Current guidelines for research ethics may not provide adequately for the nuances of 'healthy individuals' and their possible vulnerabilities. In the context of neurodegenerative disease, the fact of being 'at risk' alters the vulnerability profile in significant ways. While healthy persons who are at risk of developing dementia may not appear to warrant placement in the research category of vulnerable participants (alongside prisoners, pregnant women and children) careful regard for the vulnerabilities that arise as a result of the intersecting circumstances of being healthy and at risk of an incurable disease are worthy of increased attention and consideration, particularly as the research effort for the increasingly prevalent disease of Alzheimer's moves forward.

Aligning dissemination and implementation science with health policies to improve children's mental health.

The prevalence of mental health problems among children (ages 0-21) in the United States remains unacceptably high and, post-COVID-19, is expected to increase dramatically. Decades of psychological knowledge about effective treatments should inform the delivery of better services. Dissemination and implementation (D&I) science has been heralded as a solution to the persistent problem of poor quality services and has, to some extent, improved our understanding of the contexts of delivery systems that implement effective practices. However, there are few studies demonstrating clear, population-level impacts of psychological interventions on children. Momentum is growing among communities, cities, states, and some federal agencies to build "health in all policies" to address broad familial, social, and economic factors known to affect children's healthy development and mental health. These health policy initiatives offer a rare opportunity to repurpose D&I science, shifting it from a primary focus on evidence-based practice implementation, to a focus on policy development and implementation to support child and family health and well-being. This shift is critical as states develop policy responses to address the health and mental health impacts of the COVID-19 pandemic on already-vulnerable families. We provide a typology for building research on D&I and children's mental health policy. (PsycInfo Database Record (c) 2020 APA, all rights reserved).