ABSTRACT
BACKGROUND/AIMS: This study assesses the proficiency of Generative Pre-trained Transformer (GPT)-4 in answering questions about complex clinical ophthalmology cases. METHODS: We tested GPT-4 on 422 Journal of the American Medical Association Ophthalmology Clinical Challenges, and prompted the model to determine the diagnosis (open-ended question) and identify the next-step (multiple-choice question). We generated responses using two zero-shot prompting strategies, including zero-shot plan-and-solve+ (PS+), to improve the reasoning of the model. We compared the best-performing model to human graders in a benchmarking effort. RESULTS: Using PS+ prompting, GPT-4 achieved mean accuracies of 48.0% (95% CI (43.1% to 52.9%)) and 63.0% (95% CI (58.2% to 67.6%)) in diagnosis and next step, respectively. Next-step accuracy did not significantly differ by subspecialty (p=0.44). However, diagnostic accuracy in pathology and tumours was significantly higher than in uveitis (p=0.027). When the diagnosis was accurate, 75.2% (95% CI (68.6% to 80.9%)) of the next steps were correct. Conversely, when the diagnosis was incorrect, 50.2% (95% CI (43.8% to 56.6%)) of the next steps were accurate. The next step was three times more likely to be accurate when the initial diagnosis was correct (p<0.001). No significant differences were observed in diagnostic accuracy and decision-making between board-certified ophthalmologists and GPT-4. Among trainees, senior residents outperformed GPT-4 in diagnostic accuracy (p≤0.001 and 0.049) and in accuracy of next step (p=0.002 and 0.020). CONCLUSION: Improved prompting enhances GPT-4's performance in complex clinical situations, although it does not surpass ophthalmology trainees in our context. Specialised large language models hold promise for future assistance in medical decision-making and diagnosis.
ABSTRACT
BACKGROUND: Access to pacemakers and defibrillators is problematic in places with limited resources. Resterilization and reuse of implantable cardiac devices obtained post mortem from patients in wealthier nations have been undertaken, but uncertainty around the risk of infection is a concern. METHODS: A multinational program was initiated in 1983 to provide tested and resterilized pacemakers and defibrillators to underserved nations; a prospective registry was established in 2003. Patients who received reused devices in this program were matched in a 1:3 ratio with control patients who received new devices implanted in Canada. The primary outcome was infection or device-related death, with mortality from other causes modeled as a competing risk. RESULTS: Resterilized devices were implanted in 1051 patients (mean [±SD] age, 63.2±18.5 years; 43.6% women) in Mexico (36.0%), the Dominican Republic (28.1%), Guatemala (26.6%), and Honduras (9.3%). Overall, 85% received pacemakers and 15% received defibrillators, with one (55.5%), two (38.8%), or three (5.7%) leads. Baseline characteristics did not differ between these patients and the 3153 matched control patients. At 2 years of follow-up, infections had occurred in 21 patients (2.0%) with reused devices and in 38 (1.2%) with new devices (hazard ratio, 1.66; 95% confidence interval, 0.97 to 2.83; P = 0.06); there were no device-related deaths. The most common implicated pathogens were Staphylococcus aureus and S. epidermidis. CONCLUSIONS: Among patients in underserved countries who received a resterilized and reused pacemaker or defibrillator, the incidence of infection or device-related death at 2 years was 2.0%, an incidence that did not differ significantly from that seen among matched control patients with new devices in Canada.
