Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis.

BACKGROUND: Large language models (LLMs) have raised both interest and concern in the academic community. They offer the potential for automating literature search and synthesis for systematic reviews but raise concerns regarding their reliability, as the tendency to generate unsupported (hallucinated) content persist. OBJECTIVE: The aim of the study is to assess the performance of LLMs such as ChatGPT and Bard (subsequently rebranded Gemini) to produce references in the context of scientific writing. METHODS: The performance of ChatGPT and Bard in replicating the results of human-conducted systematic reviews was assessed. Using systematic reviews pertaining to shoulder rotator cuff pathology, these LLMs were tested by providing the same inclusion criteria and comparing the results with original systematic review references, serving as gold standards. The study used 3 key performance metrics: recall, precision, and F1-score, alongside the hallucination rate. Papers were considered "hallucinated" if any 2 of the following information were wrong: title, first author, or year of publication. RESULTS: In total, 11 systematic reviews across 4 fields yielded 33 prompts to LLMs (3 LLMs×11 reviews), with 471 references analyzed. Precision rates for GPT-3.5, GPT-4, and Bard were 9.4% (13/139), 13.4% (16/119), and 0% (0/104) respectively (P<.001). Recall rates were 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4, with Bard failing to retrieve any relevant papers (P<.001). Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001). Further analysis of nonhallucinated papers retrieved by GPT models revealed significant differences in identifying various criteria, such as randomized studies, participant criteria, and intervention criteria. The study also noted the geographical and open-access biases in the papers retrieved by the LLMs. CONCLUSIONS: Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers. The high occurrence of hallucinations in LLMs highlights the necessity for refining their training and functionality before confidently using them for rigorous academic purposes.

Hip and knee arthroplasty in one surgical session: early morbi-mortality study.

INTRODUCTION: Bilateral prosthetic hip or knee replacement in one surgical session is a procedure that has been widely validated in the literature, whereas hip and knee replacement in one surgical session remains poorly documented. This study reports on the results of these procedures by analyzing early post-operative complications in a retrospective multicenter study. MATERIAL AND METHODS: Between 2009 and 2023, 51 patients underwent hip and knee replacement surgery in a single surgical session at 4 French centers. They were 24 men and 27 women, with a mean age of 68.8 years (36-87); 7 patients were ASA 1, 30 were ASA 2 and 14 ASA 3. Prosthetic hip replacement was always performed first, associated 33 times with the ipsi-lateral knee and 18 times with the contralateral knee. All early complications, within the first 90 days post-operatively, were recorded: death, phlebitis, pulmonary embolism, myocardial infarction, surgical site infection (SSI), knee stiffness treated by mobilization under general anesthesia, urinary tract infection, acute urine retention or any other adverse event related to care. Transfusion rates were also reported. RESULTS: The rate of early complications was 9.8% (5/51). No deaths, no phlebitis, no pulmonary embolism and no SSI were observed. Complications included one myocardial infarction, one urinary tract infection, one superficial infection, one haematoma treated by surgical evacuation and one recurrent instability requiring revision surgery (hip arthroplasty). The transfusion rate was 17.6% (9/51). The complication rate of ASA 3 patients was higher than that of ASA 1 and 2 patients, while there was no difference related to age or BMI. DISCUSSION: Our results confirm the feasibility of single-stage hip and knee replacement, with a low complication rate in ASA 1 and 2 patients. This study adds to the few published works on the subject and reports comparable results. The small sample size and the heterogeneity of patients and centers limit the scope of the results, these limitations being relative to the volume expected for a rare procedure. CONCLUSION: Single-session hip and knee arthroplasty should be reserved for patients selected according to comorbidities: ASA score, age and body mass index. ASA 3 patients have a higher risk of complications. LEVEL OF EVIDENCE: IV; retrospective.

Autophagy markers are decreased in bone of osteoporotic patients: a monocentric comparative study.

BACKGROUND: Osteoporosis (OP) is a pathology characterized by bone fragility affecting 30% of postmenopausal women, mainly due to estrogen deprivation and increased oxidative stress. An autophagy involvement is suspected in OP pathogenesis but a definitive proof in humans remains to be obtained. METHODS: Postmenopausal women hospitalized for femoral neck fracture (OP group) or total hip replacement (Control group) were enrolled using very strict exclusion criteria. Western blot was used to analyze autophagy level. RESULTS: The protein expression level of the autophagosome marker LC3-II was significantly decreased in bone of OP patients relative to the control group. In addition, the protein expression of the hormonally upregulated neu-associated kinase (HUNK), which is upregulated by female hormones and promotes autophagy, was also significantly reduced in bone of the OP group. CONCLUSIONS: These results demonstrate for the first time that postmenopausal OP patients have a deficit in bone autophagy level and suggest that HUNK could be the factor linking estrogen loss and autophagy decline. CLINICAL TRIAL REGISTRATION NUMBER: ClinicalTrials.gov Identifier: NCT03175874, 2/6/2017.