Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis.

BACKGROUND: Large language models (LLMs) have raised both interest and concern in the academic community. They offer the potential for automating literature search and synthesis for systematic reviews but raise concerns regarding their reliability, as the tendency to generate unsupported (hallucinated) content persist. OBJECTIVE: The aim of the study is to assess the performance of LLMs such as ChatGPT and Bard (subsequently rebranded Gemini) to produce references in the context of scientific writing. METHODS: The performance of ChatGPT and Bard in replicating the results of human-conducted systematic reviews was assessed. Using systematic reviews pertaining to shoulder rotator cuff pathology, these LLMs were tested by providing the same inclusion criteria and comparing the results with original systematic review references, serving as gold standards. The study used 3 key performance metrics: recall, precision, and F1-score, alongside the hallucination rate. Papers were considered "hallucinated" if any 2 of the following information were wrong: title, first author, or year of publication. RESULTS: In total, 11 systematic reviews across 4 fields yielded 33 prompts to LLMs (3 LLMs×11 reviews), with 471 references analyzed. Precision rates for GPT-3.5, GPT-4, and Bard were 9.4% (13/139), 13.4% (16/119), and 0% (0/104) respectively (P<.001). Recall rates were 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4, with Bard failing to retrieve any relevant papers (P<.001). Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001). Further analysis of nonhallucinated papers retrieved by GPT models revealed significant differences in identifying various criteria, such as randomized studies, participant criteria, and intervention criteria. The study also noted the geographical and open-access biases in the papers retrieved by the LLMs. CONCLUSIONS: Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers. The high occurrence of hallucinations in LLMs highlights the necessity for refining their training and functionality before confidently using them for rigorous academic purposes.

Anti-rituximab antibodies demonstrate neutralizing capacity, associate with lower circulating drug levels and earlier relapse in lupus.

OBJECTIVES: High rates of anti-drug antibodies (ADA) to rituximab have been demonstrated in patients undergoing treatment for SLE. However, little is known with regard to their long-term dynamics, impact on drug kinetics and subsequent implications for treatment response. In this study, we aimed to evaluate ADA persistence over time, impact on circulating drug levels, assess clinical outcomes and whether they are capable of neutralizing rituximab. METHODS: Patients with SLE undergoing treatment with rituximab were recruited to this study (n = 35). Serum samples were collected across a follow-up period of 36 months following treatment (n = 114). Clinical and laboratory data were collected pre-treatment and throughout follow-up. ADA were detected via electrochemiluminescent immunoassays. A complement dependent cytotoxicity assay was used to determine neutralizing capacity of ADA in a sub-cohort of positive samples (n = 38). RESULTS: ADA persisted over the 36-month study period in 64.3% of patients undergoing treatment and titres peaked earlier and remained higher in those who had previously been treated with rituximab when compared with than those who were previously treatment naive. ADA-positive samples had a significantly lower median drug level until six months post rituximab infusion (P = 0.0018). Patients with persistent ADA positivity showed a significant early improvement in disease activity followed by increased rates of relapse. In vitro analysis confirmed the neutralizing capacity of ADA to rituximab. CONCLUSIONS: ADA to rituximab were common and persisted over the 36-month period of this study. They associated with earlier drug elimination, an increased rate of relapse and demonstrated neutralizing capacity in vitro.

Sphingosine-1 Phosphate Receptor Modulators Increase In Vitro Melanoma Cell Line Proliferation at Therapeutic Doses Used in Patients with Multiple Sclerosis.

INTRODUCTION: S1P1 receptor modulators (S1P1-RM) are oral disease-modifying therapies (DMTs) for multiple sclerosis (MS). Several authorities have raised doubts that S1P1-RM are responsible for an increased risk of melanoma in patients with MS. We studied the in vitro effects of S1P1-RM on different melanoma cell lines to compare the effect of available S1P1-RM on the proliferation of human melanoma cells. METHODS: Four S1P1-RM were studied which are currently approved for managing MS, namely fingolimod (Gilenya®), siponimod (Mayzent®), ozanimod (Zeposia®), and ponesimod (Ponvory®). We tested these four drugs at different concentrations, including therapeutic doses (0.5, 1.6, 5.5, 18, and 60 µM), on human melanoma cell lines (501Mel cells, 1205LU cells, and M249R cells) to analyze in vitro cell proliferation monitored with the IncuCyte ZOOM live cell microscope (Essen Bioscience). RESULTS: At therapeutic doses, median confluence increased overall for all lineages: + 122% for ozanimod (p < 0.001), + 71% for ponesimod (p < 0.001), + 67% for siponimod (NS), and + 41% for fingolimod (p = 0.094). Ozanimod- and ponesimod-treated cells increased confluency in 501Mel, 1205LU, and M249R cell lines (p < 0.001). CONCLUSION: These data suggest an increased proliferation of various melanoma cell lines with S1P1-RM treatments used at therapeutic concentrations for patients with MS and should raise the question of increased dermatologic surveillance.

Low baseline IFN-γ response could predict hospitalization in COVID-19 patients.

The SARS-CoV-2 infection has spread rapidly around the world causing millions of deaths. Several treatments can reduce mortality and hospitalization. However, their efficacy depends on the choice of the molecule and the precise timing of its administration to ensure viral clearance and avoid a deleterious inflammatory response. Here, we investigated IFN-γ, assessed by a functional immunoassay, as a predictive biomarker for the risk of hospitalization at an early stage of infection or within one month prior to infection. Individuals with IFN-γ levels below 15 IU/mL were 6.57-times more likely to be hospitalized than those with higher values (p<0.001). As confirmed by multivariable analysis, low IFN-γ levels, age >65 years, and no vaccination were independently associated with hospitalization. In addition, we found a significant inverse correlation between low IFN-γ response and high level of IL-6 in plasma (Spearman's rho=-0.38, p=0.003). Early analysis of the IFN-γ response in a contact or recently infected subject with SARS-CoV-2 could predict hospitalization and thus help the clinician to choose the appropriate treatment avoiding severe forms of infection and hospitalization.

Memory B Cells Predict Relapse in Rituximab-Treated Myasthenia Gravis.

Myasthenia gravis can be efficiently treated with rituximab but there is no consensus regarding administration and dose schedules in this indication. No marker has yet been described to predict the clinical relapse of patients. Our objective was to identify the B cell subpopulations predicting clinical relapse in patients suffering from generalized myasthenia gravis and treated with rituximab. Clinical and biological data of 34 patients followed between 2016 and 2019 were prospectively collected every 3 months. Using multiparameter flow cytometry, we assessed the percentage in leucocytes of lymphocytes and several B cell subpopulations measured in residual disease conditions. CD19+ were also measured in non-residual disease conditions. Clinical examinations were performed by neurologists using the Osserman score. Clinical relapse occurred in 14 patients (41%). No patients required ICU or ventilatory assistance. The mean improvement of the Osserman score was 17.18 (3-45) after the first rituximab treatment (p < 0.0001). The mean delay between the first rituximab maintenance cycle and clinical relapse was 386.8 days. At the time of relapse, CD27+ increased (p = 0.0006) with AUC = 0.7654, while CD19+ did not. At a threshold of 0.01%, the sensitivity and specificity of CD19+CD27+ were 75.8% and 72.8%, respectively, and the positive and negative predictive values were 28.0% and 95.6%, respectively. The percentage of memory B cells in whole blood cells can accurately predict clinical relapse in myasthenia gravis patients treated with rituximab. This monitoring allows physicians to tailor rituximab administration and to decrease the number of infusions over time.

Baseline and early functional immune response is associated with subsequent clinical outcomes of PD-1 inhibition therapy in metastatic melanoma patients.

BACKGROUND: Despite significant progress with antiprogrammed cell death protein 1 (PD-1) therapy, a substantial fraction of metastatic melanoma patients show upfront therapy resistance. Biomarkers for outcome are missing and the association of baseline immune function and clinical outcome remains to be determined. We assessed the in vitro nonspecific stimulation of immune response at baseline and during anti-PD-1 therapy for metastatic melanoma. METHODS: Previously untreated metastatic melanoma patients received nivolumab and radiotherapy as part of the multicentric phase II trial NIRVANA (NCT02799901). The levels of Th1, Th2 and Th17 cytokines on in vitro non-specific stimulation of innate and adaptive immune cells were measured in patient sera before treatment, and at week 2 and week 6 after the beginning of the treatment, and correlated with tumorous response, progression-free survival (PFS) and occurrence of immune-related adverse events (irAEs). The results in melanoma patients were compared with those of a cohort of 9 sex and age-matched healthy donors. RESULTS: Seventeen patients were enrolled in this ancillary study. Median follow-up was 16 months (2.2-28.4). The 12-month PFS rate was 67.7%. The incidence of irAEs of any grade was 58.8%. Without in vitro stimulation no differences in cytokines levels were observed between responders and non-responders. On in vitro stimulation, metastatic patients had lower Th1 cytokine levels than healthy donors at baseline for tumor necrosis factor-α and interferon-γ (IFN-γ) (1136 pg/mL vs 5558 pg/mL, p<0.0001; and 3894 pg/mL vs 17 129 pg/mL, p=0.02, respectively). Responders exhibited increasing cytokine levels from baseline to week 6. Non-responders had lower interleukin 17A (IL-17A) levels at baseline than responders (7 pg/mL vs 32 pg/mL, p=0.03), and lower IFN-γ levels at week 6 (3.3 ng/mL vs 14.5 ng/mL, p=0.03). A lower level of IL-17A at week 2 and a lower level of IFN-γ at week 6 correlated with worse PFS (p=0.04 and p=0.04 respectively). At baseline, patients who developed irAEs had higher IL-6 levels (19.3 ng/mL vs 9.2 ng/mL, p=0.03) and higher IL-17A levels (52.5 pg/mL vs 2.5 pg/mL, p=0.009) than those without irAEs. CONCLUSIONS: Our findings indicate that cytokine levels after in vitro non-specific stimulation could be a promising biomarker to predict the outcome of PD-1 inhibition therapy.