Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis.

BACKGROUND: Large language models (LLMs) have raised both interest and concern in the academic community. They offer the potential for automating literature search and synthesis for systematic reviews but raise concerns regarding their reliability, as the tendency to generate unsupported (hallucinated) content persist. OBJECTIVE: The aim of the study is to assess the performance of LLMs such as ChatGPT and Bard (subsequently rebranded Gemini) to produce references in the context of scientific writing. METHODS: The performance of ChatGPT and Bard in replicating the results of human-conducted systematic reviews was assessed. Using systematic reviews pertaining to shoulder rotator cuff pathology, these LLMs were tested by providing the same inclusion criteria and comparing the results with original systematic review references, serving as gold standards. The study used 3 key performance metrics: recall, precision, and F1-score, alongside the hallucination rate. Papers were considered "hallucinated" if any 2 of the following information were wrong: title, first author, or year of publication. RESULTS: In total, 11 systematic reviews across 4 fields yielded 33 prompts to LLMs (3 LLMs×11 reviews), with 471 references analyzed. Precision rates for GPT-3.5, GPT-4, and Bard were 9.4% (13/139), 13.4% (16/119), and 0% (0/104) respectively (P<.001). Recall rates were 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4, with Bard failing to retrieve any relevant papers (P<.001). Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001). Further analysis of nonhallucinated papers retrieved by GPT models revealed significant differences in identifying various criteria, such as randomized studies, participant criteria, and intervention criteria. The study also noted the geographical and open-access biases in the papers retrieved by the LLMs. CONCLUSIONS: Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers. The high occurrence of hallucinations in LLMs highlights the necessity for refining their training and functionality before confidently using them for rigorous academic purposes.

The reliability of the Neer classification for proximal humerus fractures: a survey of orthopedic shoulder surgeons.

Background: The Neer classification is among the most widely used systems to describe proximal humerus fractures (PHF) despite the poor interobserver agreement. The purpose of this study was to verify whether or not blinded shoulder surgeons and trainees agree with the authors of articles published in the highest impact-factor orthopedic journals. Methods: All articles regarding PHF published between 2017 and 2019 in the top 10 orthopedic journals as rated by impact factor were searched. Articles were included if the authors used the Neer classification to describe at least 1 PHF in the figures. Figures were extracted without the legend, and X-rays ± computed tomography scan images were included when available. An international survey was conducted among 138 shoulder surgeons who were asked to record the Neer classification for each de-identified radiograph in the publications. The type of fracture mentioned in the legend of the published figure was considered as the gold standard. Results: Survey participants agreed with the published article authors in 55% of cases overall (range 6%-96%, n = 35). The most common response disagreed with the article authors in 13 cases (37%), underestimating the number of parts in 11 of 13 cases. The interobserver agreement between the 138 responders was fair (k = 0.296). There was an association between the percentage of concordant answers and greater experience (number of years of shoulder surgery practice) of the responders (P = .0023). The number of parts, the number or type of available imaging modalities, and the geographic origin of participants did not influence the agreement between responders and authors. Discussion: In more than one-third of cases, specialized shoulder surgeons disagree with article authors when interpreting the Neer classification of de-identified images of PHF in published manuscripts. Morphologic classification of PHF as the sole basis for treatment algorithms and surgical success should be scrutinized.

Short-term outcomes and survival of pyrocarbon hemiarthroplasty in the young arthritic shoulder.

HYPOTHESIS: The purpose was to report the short-term outcomes and survival of hemiarthroplasty with a pyrocarbon head (HA-PYC) for the treatment of shoulder osteoarthritis in patients aged ≤ 60 years. We hypothesized that HA-PYC could be an alternative to hemi-metal (avoiding the risk of rapid glenoid erosion) and total shoulder arthroplasty (TSA) (avoiding the risk of rapid glenoid loosening) in an active patient population. METHODS: Sixty-four consecutive patients (mean age, 53 years) who underwent HA-PYC for glenohumeral osteoarthritis were included. The primary outcome was revision to TSA or reverse shoulder arthroplasty. Secondary outcome measures included functional outcome scores; return to work and sports; and radiographic evaluation of humeral reconstruction quality using the "circle method" of Mears, as well as glenoid erosion severity and progression using the Sperling classification. Patients were reviewed and underwent radiography at an average follow-up of 33 months (range, 24-60 months). RESULTS: At last follow-up, the rate of survival of the HA-PYC prosthesis was 92%. Revision was performed in 5 patients, with a mean delay of 24 months (range, 15-37 months): 1 conversion to TSA and 4 conversions to reverse shoulder arthroplasty. The Constant score and Subjective Shoulder Value increased from 36 points (range, 26-50 points) to 75 points (range, 69-81 points) and from 35% (range, 20%-50%) to 80% (range, 75%-90%), respectively (P < .001). Postoperatively, 91% of the patients (42 of 46) returned to work and 88% (15 of 17) returned to sport. The severity of preoperative and postoperative glenoid wear (Sperling grade 3 or 4) had no influence on the functional results. Patients who underwent associated concentric glenoid reaming (n = 23) had similar Constant scores and Subjective Shoulder Values (P = .95) to other patients and did not show more progression of glenoid wear. Nonanatomic reconstruction of the proximal humerus (center of rotation of the prosthesis > 3 mm from the anatomic center) occurred in 29% (18 of 62 patients) and was associated with significantly lower functional and subjective results, more complications (subscapularis insufficiency and/or symptomatic glenoid erosion), and a higher risk of revision. The additional 1.5-mm thickness of the metal disc under the pyrocarbon head was found to be the main reason for oversizing of the prosthetic head. CONCLUSION: HA-PYC is a reliable procedure to treat shoulder osteoarthritis and allows return to work and sports in a young (≤60 years) and active patient population. The severity of glenoid bone erosion or the association with glenoid reaming does not affect functional outcomes and failure risk. By contrast, nonanatomic reconstruction of the proximal humerus after HA-PYC (because of humeral head oversizing) occurred in one-third of the cases and is associated with lower functional outcomes, as well as higher risks of complications and revision.

A Population Pharmacokinetic Analysis of Continuous Infusion of Cloxacillin during Staphylococcus aureus Bone and Joint Infections.

Intravenous administration of antibiotics is recommended during the early phase of methicillin-susceptible S. aureus (MSSA) bone and joint infection (BJI). We sought to compare the plasma concentrations of cloxacillin administered alternately by continuous and intermittent infusion (CI and ItI) in patients with MSSA BJI. In this prospective crossover trial, patients were randomly assigned to receive either 3 days of CI (two 75-mg/kg 12-h cloxacillin infusions per day) and then 3 days of ItI (four 37.5-mg/kg 1-h cloxacillin infusions per day) or vice versa. The drug concentration measurement was performed on day 3 of each type of administration at 1, 6, and 11 h and at 1, 2, 3, 4, and 6 h after the beginning of CI and ItI, respectively. We used the nonparametric algorithm NPAG to estimate population pharmacokinetic (PK) parameters. The final model was used to perform pharmacokinetic/pharmacodynamic (PK/PD) simulations and calculate the probabilities of target attainment (PTA) for several ItI and CI dosing regimens. We considered two PK/PD targets of time spent above the MIC for free cloxacillin concentrations (fT>MIC): 50 and 100%. Eighty-four concentrations from 11 patients were analyzed. A two-compartment model adequately described the data. ItI with q6h regimens and short 1-h infusions of 2,000 or 3,000 mg were associated with low PTA, even for the low target (50% fT>MIC) while 3-h infusions and continuous infusions (6 to 12 g/day) were associated with a PTA of >90% for an MIC up to 0.5 mg/liter. These results support the use of prolonged or continuous infusion of cloxacillin in patients with BJI.

Reverse shoulder-allograft prosthesis composite, with or without tendon transfer, for the treatment of severe proximal humeral bone loss.

BACKGROUND: The treatment of severe proximal humeral bone loss (PHBL) secondary to tumor resection or failed arthroplasty is challenging. We evaluated the outcomes and complications of reconstruction with reverse shoulder-allograft prosthesis composite (RS-APC), performed with or without tendon transfer. METHODS: An RS-APC procedure was performed in 25 consecutive patients with severe PHBL (>4 cm): 12 after failed reverse shoulder arthroplasty, 5 after failed hemiarthroplasty for fracture, 6 after failed mega-tumor prosthesis placement, and 2 after tumor resection. The median length of humeral bone loss or resection was 8 cm (range, 5-23 cm). Humeral bone graft fixation was obtained with a long monobloc reverse stem and a "mirror step-cut osteotomy," without plate fixation. Nine infected shoulders underwent a 2-stage operation with a temporary cement spacer. In addition, 9 patients (36%) underwent an associated L'Episcopo procedure. The median follow-up duration was 4 years (range, 2-11 years). RESULTS: Overall, 76% of patients (19 of 25) were satisfied. In 8 patients (32%), a reoperation was needed. At last follow-up, we observed incorporation at the allograft-host junction in 96% of the cases (24/25); partial graft resorption occurred in 3 cases and severe in 1. The median adjusted Constant score was 53% (range 18-105); Subjective Shoulder Value, 50% (range 10%-95%). Additional tendon transfers significantly improved active external rotation (20° vs. 0°, P < .001) and forward elevation (140° vs. 90°, P = .045). CONCLUSIONS: (1) Shoulder reconstruction with RS-APC provides acceptable shoulder function and high rates of graft survival and healing. (2) Additional L'Episcopo tendon transfer (when technically possible) improves active shoulder motion. (3) The use of a long monobloc (cemented or uncemented) humeral reverse stem with mirror step-cut osteotomy provides a high rate of graft-host healing, as well as a limited rate of graft resorption, and precludes the need for additional plate fixation. (4) Although rewarding, this reconstructive surgery is complex with a high risk of complications and reoperations. The main advantages of using an allograft with a reverse shoulder arthroplasty (compared with other reconstruction options) are that this type of reconstruction (1) allows restoration of the bone stock, thus improving prosthesis fixation and stability, and (2) gives the possibility to perform a tendon transfer by fixing the tendons on the bone graft to improve shoulder motion.

The 14 July 2016 terrorist attack in Nice: The experience of orthopaedic surgeons.

BACKGROUND: On 14 July 2016, a terrorist drove a truck through the crowd on the Promenade des Anglais in Nice, France, killing 87 people and injuring 458. The objective of this study was to evaluate the management strategy used to handle the osteo-articular injuries caused by this attack. HYPOTHESIS: The management strategy used ensured that open fractures were treated within 6hours. MATERIAL AND METHOD: This single-centre retrospective study included all victims of the attack admitted to the Pasteur 2 Hospital in Nice, France, for osteo-articular injuries, and treated between 14 and 31 July 2016. The following data were collected for each patient: age, sex, type of injury, Injury Severity Score (ISS), whether the damage control orthopaedics (DCO) or early total care (ETC) approach was followed, time from injurytotreatment, operative time, and surgical revisions. The primary outcome measure was the injury to treatment time for each lesion. RESULTS: Of the 182 patients admitted to the emergency department, 32 required admission for osteo-articular injuries, including 18 with severe injuries (ISS>15) and 11 with multiple fractures. Their injuries were of the type seen in traffic accidents. Of the 87 fractures, 45% involved the lower limbs and 25% were open fractures. Surgery was performed in 14 patients on the first night (14 to 15 July) and in 19 patients overall. The approach was DCO in 12 and ETC in 7 of these 19 patients. All lesions were managed within recommended time intervals, including the 21 open fractures and 2 closed femoral shaft fractures. DISCUSSION: Injury-to-surgery time complied with recommendations in all cases. In 25% of cases, ETC would have been feasible during the mass influx of patients without hospital capacity saturation. LEVEL OF EVIDENCE: IV, retrospective observational study.

Surgical management of isolated superficial femoral artery degenerative aneurysms.

OBJECTIVE: To investigate the mode of presentation, diagnosis, association with other aneurysms, operative management, and outcomes of superficial femoral artery (SFA) aneurysms. METHODS: Records of all patients who underwent surgery for isolated, true SFA aneurysms (not due to infection, vasculitis, or tissue disorders) from 2002 to 2012 in two European centers were retrospectively analyzed. Demographic (sex, age), clinical (cardiovascular risk factors, location of the aneurysm, symptoms, presentation, emergency setting), surgical and radiological data (diameter, surgical technique, runoff vessels patency, presence of aneurysms elsewhere) were obtained for analysis. Follow-up was undertaken with clinical and ultrasound examinations at 1 month, 3 months, 6 months, 12 months, and yearly thereafter. The patency of the graft and the status of the anastomoses and inflow and outflow vessels were assessed. Main end points were represented by 30 days and long-term mortality and amputation-free survival. RESULTS: A total of 27 cases of SFA aneurysm were analyzed. Mean age at operation was 78 years ± 8.5. At presentation, SFA aneurysms were often symptomatic (rupture was present in 7/27 cases and acute distal ischemia in 6/27 cases), large (mean diameter, 54 mm ± 33.1 mm), bilateral (38% of the cases), and associated with aneurysms elsewhere (84%). Sixteen patients underwent resection of the aneurysm and polytetrafluoroethylene interposition graft, seven patients exclusion of the aneurysm with a femoropopliteal bypass (autogenous bypass in five cases, prosthetic in two), three patients simple ligation, and one patient underwent primary amputation. Mean follow-up was 41.47 months (range, 0.43-128.67 months). Early (<30 days) mortality and amputation rate were 4% and 7%, respectively. Estimated 5-year survival, limb salvage, and graft patency rates were 62%, 88%, and 85%, respectively. CONCLUSIONS: Degenerative aneurysms of the SFA display peculiar characteristics (in terms of clinical onset, diagnostic timing, and clinical behavior) so that they differ from other peripheral aneurysms. In fact, they often grow to reach a considerable diameter before medical attention is sought, presenting with rupture or ischemia at diagnosis. SFA aneurysms preferentially affect elderly men and are often associated with aneurysms elsewhere. However, despite their rarity, the treatment is usually feasible, and long-term outcomes are good.