Integrating human expertise & automated methods for a dynamic and multi-parametric evaluation of large language models' feasibility in clinical decision-making.

BACKGROUND: Recent enhancements in Large Language Models (LLMs) such as ChatGPT have exponentially increased user adoption. These models are accessible on mobile devices and support multimodal interactions, including conversations, code generation, and patient image uploads, broadening their utility in providing healthcare professionals with real-time support for clinical decision-making. Nevertheless, many authors have highlighted serious risks that may arise from the adoption of LLMs, principally related to safety and alignment with ethical guidelines. OBJECTIVE: To address these challenges, we introduce a novel methodological approach designed to assess the specific feasibility of adopting LLMs within a healthcare area, with a focus on clinical nursing, evaluating their performance and thereby directing their choice. Emphasizing LLMs' adherence to scientific advancements, this approach prioritizes safety and care personalization, according to the "Organization for Economic Co-operation and Development" frameworks for responsible AI. Moreover, its dynamic nature is designed to adapt to future evolutions of LLMs. METHOD: Through integrating advanced multidisciplinary knowledge, including Nursing Informatics, and aided by a prospective literature review, seven key domains and specific evaluation items were identified as follows:A Peer Review by experts in Nursing and AI was performed, ensuring scientific rigor and breadth of insights for an essential, reproducible, and coherent methodological approach. By means of a 7-point Likert scale, thresholds are defined in order to classify LLMs as "unusable", "usable with high caution", and "recommended" categories. Nine state of the art LLMs were evaluated using this methodology in clinical oncology nursing decision-making, producing preliminary results. Gemini Advanced, Anthropic Claude 3 and ChatGPT 4 achieved the minimum score of the State of the Art Alignment & Safety domain for classification as "recommended", being also endorsed across all domains. LLAMA 3 70B and ChatGPT 3.5 were classified as "usable with high caution." Others were classified as unusable in this domain. CONCLUSION: The identification of a recommended LLM for a specific healthcare area, combined with its critical, prudent, and integrative use, can support healthcare professionals in decision-making processes.

Transition from intravenous to subcutaneous biological therapies in inflammatory bowel disease: An online survey of patients.

BACKGROUND: The transition from in-hospital intravenous administration to subcutaneous therapies to treat inflammatory bowel disease (IBD) can raise some concerns among patients due to the self-administration concerns, the management of potential side effects and the overall worries related to a change of treatment. This study aimed at evaluating patients' opinion about the switch from intravenous to subcutaneous formulations and their knowledge on new available therapeutic options. METHODS: We conducted a survey using a questionnaire prepared by a team of gastroenterologists and nurses working at the IBD unit. It consists of 31 items and has been divided into four sections: descriptive, commitment, knowledge and passage mode opinion. The questions were formulated in Italian and conceived according to daily consultations with patients in everyday practice, without any previous piloting or specific medical literature reference. The survey was administered to consecutive IBD patients in intravenous biological treatment; patients currently or previously treated with subcutaneous therapy were excluded. RESULTS: Four hundred questionnaires were distributed to participants. As many as 311 patients (77.7%) completed the survey, while the remaining were excluded from the analysis; 155 (49.8%) patients were favorable to switch from intravenous to subcutaneous therapy, while only 78 (25.1%) disagreed. In univariate and multi-variate analysis, the approval rate for home therapy was significantly associated with the distance from the IBD center and work/family/personal commitments. Surprisingly, only a quarter of the IBD patients knew that almost all available therapeutic agents have a subcutaneous administration route. Regarding patients' opinion on the efficacy of subcutaneous administration of biological agents compared to intravenous drugs, 194 (63%) had no definite idea, while 44 (14%) believed that the effectiveness could be reduced. CONCLUSION: The transition from in-hospital to subcutaneous therapeutic management of biological therapy at home was generally viewed favorably by patients, especially if they have commitments or were residents far from the IBD center.