Performance of Large Language Models in Patient Complaint Resolution: Web-Based Cross-Sectional Survey.

BACKGROUND: Patient complaints are a perennial challenge faced by health care institutions globally, requiring extensive time and effort from health care workers. Despite these efforts, patient dissatisfaction remains high. Recent studies on the use of large language models (LLMs) such as the GPT models developed by OpenAI in the health care sector have shown great promise, with the ability to provide more detailed and empathetic responses as compared to physicians. LLMs could potentially be used in responding to patient complaints to improve patient satisfaction and complaint response time. OBJECTIVE: This study aims to evaluate the performance of LLMs in addressing patient complaints received by a tertiary health care institution, with the goal of enhancing patient satisfaction. METHODS: Anonymized patient complaint emails and associated responses from the patient relations department were obtained. ChatGPT-4.0 (OpenAI, Inc) was provided with the same complaint email and tasked to generate a response. The complaints and the respective responses were uploaded onto a web-based questionnaire. Respondents were asked to rate both responses on a 10-point Likert scale for 4 items: appropriateness, completeness, empathy, and satisfaction. Participants were also asked to choose a preferred response at the end of each scenario. RESULTS: There was a total of 188 respondents, of which 115 (61.2%) were health care workers. A majority of the respondents, including both health care and non-health care workers, preferred replies from ChatGPT (n=164, 87.2% to n=183, 97.3%). GPT-4.0 responses were rated higher in all 4 assessed items with all median scores of 8 (IQR 7-9) compared to human responses (appropriateness 5, IQR 3-7; empathy 4, IQR 3-6; quality 5, IQR 3-6; satisfaction 5, IQR 3-6; P<.001) and had higher average word counts as compared to human responses (238 vs 76 words). Regression analyses showed that a higher word count was a statistically significant predictor of higher score in all 4 items, with every 1-word increment resulting in an increase in scores of between 0.015 and 0.019 (all P<.001). However, on subgroup analysis by authorship, this only held true for responses written by patient relations department staff and not those generated by ChatGPT which received consistently high scores irrespective of response length. CONCLUSIONS: This study provides significant evidence supporting the effectiveness of LLMs in resolution of patient complaints. ChatGPT demonstrated superiority in terms of response appropriateness, empathy, quality, and overall satisfaction when compared against actual human responses to patient complaints. Future research can be done to measure the degree of improvement that artificial intelligence generated responses can bring in terms of time savings, cost-effectiveness, patient satisfaction, and stress reduction for the health care system.

Factors reducing inappropriate attendances to emergency departments before and during the COVID-19 pandemic: A multicentre study.

INTRODUCTION: Inappropriate attendances (IAs) to emergency departments (ED) create an unnecessary strain on healthcare systems. With decreased ED attendance during the COVID-19 pandemic, this study postulates that there are less IAs compared to before the pandemic and identifies factors associated with IAs. METHODS: We performed a retrospective review of 29,267 patient presentations to a healthcare cluster in Singapore from 7 April 2020 to 1 June 2020, and 36,370 patients within a corresponding period in 2019. This time frame coincided with local COVID-19 lockdown measures. IAs were defined as patient presentations with no investigations required, with patients eventually discharged from the ED. IAs in the 2020 period during the pandemic were compared with 2019. Multivariable logistic regression was performed to identify factors associated with IAs. RESULTS: There was a decrease in daily IAs in 2020 compared to 2019 (9.91±3.06 versus 24.96±5.92, P<0.001). IAs were more likely with self-referrals (adjusted odds ratio [aOR] 1.58, 95% confidence interval [CI] 1.50-1.66) and walk-ins (aOR 4.96, 95% CI 4.59-5.36), and those diagnosed with non-specific headache (aOR 2.08, 95% CI 1.85-2.34), or non-specific low back pain (aOR 1.28, 95% CI 1.15-1.42). IAs were less likely in 2020 compared to 2019 (aOR 0.67, 95% CI 0.65-0.71) and older patients (aOR 0.79 each 10 years, 95% CI 0.78-0.80). CONCLUSION: ED IAs decreased during COVID-19. The pandemic has provided a unique opportunity to examine factors associated with IAs.