Assessing ChatGPT 4.0's test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports.

While there is data assessing the test performance of artificial intelligence (AI) chatbots, including the Generative Pre-trained Transformer 4.0 (GPT 4) chatbot (ChatGPT 4.0), there is scarce data on its diagnostic accuracy of clinical cases. We assessed the large language model (LLM), ChatGPT 4.0, on its ability to answer questions from the United States Medical Licensing Exam (USMLE) Step 2, as well as its ability to generate a differential diagnosis based on corresponding clinical vignettes from published case reports. A total of 109 Step 2 Clinical Knowledge (CK) practice questions were inputted into both ChatGPT 3.5 and ChatGPT 4.0, asking ChatGPT to pick the correct answer. Compared to its previous version, ChatGPT 3.5, we found improved accuracy of ChatGPT 4.0 when answering these questions, from 47.7 to 87.2% (p = 0.035) respectively. Utilizing the topics tested on Step 2 CK questions, we additionally found 63 corresponding published case report vignettes and asked ChatGPT 4.0 to come up with its top three differential diagnosis. ChatGPT 4.0 accurately created a shortlist of differential diagnoses in 74.6% of the 63 case reports (74.6%). We analyzed ChatGPT 4.0's confidence in its diagnosis by asking it to rank its top three differentials from most to least likely. Out of the 47 correct diagnoses, 33 were the first (70.2%) on the differential diagnosis list, 11 were second (23.4%), and three were third (6.4%). Our study shows the continued iterative improvement in ChatGPT's ability to answer standardized USMLE questions accurately and provides insights into ChatGPT's clinical diagnostic accuracy.

Chest vs. neck anastomotic leak post esophagectomy for malignancy: rate, predictors, and outcomes.

Background: Anastomotic leak is a major contributor to comorbidity and mortality following esophagectomy. We sought to assess rate and predictors of leak after esophagectomy and compare outcomes of chest versus neck anastomotic leaks. Methods: A retrospective review was performed utilizing National-Surgical-Quality-Improvement-Program data from 2016-2019 for patients undergoing esophagectomy for malignancy. Preoperative characteristics and postoperative outcomes were compared. Patients were classified into two groups: Ivor Lewis esophagectomy [ILE, chest leak (CL)] and transhiatal esophagectomy (THE)/McKeown esophagectomy [ME, neck leak (NL)]. Multivariable regression models were constructed to determine predictors of each type of leak and postoperative complications. Results: A total of 1,665 patients underwent esophagectomy with 14.1% reported post-operative leak, 61% of patients underwent ILE while 39% underwent THE or ME. Of patients who underwent ILE, 13.8% had CL with complications including significantly higher length of stay and mortality compared to patients without leak. Independent predictors of CL included: diabetes, hypertension, advanced disease stage, chronic steroid use, and operative time. Ninety-five patients (14.6%) who underwent either THE or ME had NL with similar complications. Diabetes, pre-operative white blood cell (WBC), and operative time were independent predictors for NL. On multivariable regression, CL was associated with greater odds of requiring intervention compared with NL. Conclusions: Post-esophagectomy CL and NL are associated with higher morbidity and mortality. Diabetes and operative time were independent predictors for both leaks while steroid use, hypertension, and advanced disease stage predicted CL. CL was associated with greater odds of needing an intervention, but contrary to conventional wisdom, was not associated with higher morbidity or mortality.