OCTess: AN OPTICAL CHARACTER RECOGNITION ALGORITHM FOR AUTOMATED DATA EXTRACTION OF SPECTRAL DOMAIN OPTICAL COHERENCE TOMOGRAPHY REPORTS.

PURPOSE: Manual extraction of spectral domain optical coherence tomography (SD-OCT) reports is time and resource intensive. This study aimed to develop an optical character recognition (OCR) algorithm for automated data extraction from Cirrus SD-OCT macular cube reports. METHODS: SD-OCT monocular macular cube reports (n = 675) were randomly selected from a single-center database of patients from 2020 to 2023. Image processing and bounding box operations were performed, and Tesseract (an OCR library) was used to develop the algorithm, OCTess. The algorithm was validated using a separate test data set. RESULTS: The long short-term memory deep learning version of Tesseract achieved the best performance. After reverifying all discrepancies between human and algorithmic data extractions, OCTess achieved accuracies of 100.00% and 99.98% in the training (n = 125) and testing (n = 550) datasets, while the human error rate was 1.11% (98.89% accuracy) and 0.49% (99.51% accuracy) in each, respectively. OCTess extracted data in 3.1 seconds, compared with 94.3 seconds per report for human evaluators. CONCLUSION: We developed an OCR and machine learning algorithm that extracted SD-OCT data with near-perfect accuracy, outperforming humans in both accuracy and efficiency. This algorithm can be used for efficient construction of large-scale SD-OCT data sets for researchers and clinicians.

VALIDATION OF M-CHARTS FOR QUANTITATIVE ASSESSMENT OF METAMORPHOPSIA FOLLOWING RHEGMATOGENOUS RETINAL DETACHMENT REPAIR.

PURPOSE: To validate the quantitative assessment of metamorphopsia in rhegmatogenous retinal detachment (RRD) using M-CHARTS by determining its correlation with subjective reporting of metamorphopsia with a validated metamorphopsia questionnaire (modified MeMoQ). METHODS: The Research Ethics Board approved a prospective observational study carried out at St. Michael's Hospital, Toronto, Canada. Patients with primary, unilateral RRD and healthy controls were included. Metamorphopsia at 3 months was assessed with modified MeMoQ and M-CHARTS. RESULTS: One hundred patients (50 with RRD, 50 controls) were included. Seventy percent (35/50) of the RRD group had metamorphopsia with M-CHARTS and 80% (40/50) with MeMoQ. The modified MeMoQ and total M-CHARTS scores were significantly higher in patients with RRD compared with controls ( P < 0.0001). Cronbach's alpha reliability coefficient was 0.934 in the RRD group. Horizontal, vertical, and total M-CHARTS scores were significantly correlated with MeMoQ scores (r s  = 0.465, P = 0.0007; r s  = 0.405, P = 0.004; r s  = 0.475, P = 0.0005, respectively). M-CHARTS was 72.7% sensitive and 94.6% specific for detection of metamorphopsia (positive score ≥ 0.2), with an area under the receiver operating characteristic curve = 0.801. A stronger correlation was found in patients who scored ≥0.2 on the M-CHARTS and reported metamorphopsia with the MeMoQ (r s  = 0.454, P = 0.001). CONCLUSION: The authors have validated M-CHARTS as a tool to quantitatively assess metamorphopsia in patients with RRD, which is significantly correlated with patient-reported outcomes using the MeMoQ. A total score of ≥0.2 with M-CHARTS was more strongly correlated with MeMoQ.

Performance of Large Language Models on Medical Oncology Examination Questions.

Importance: Large language models (LLMs) recently developed an unprecedented ability to answer questions. Studies of LLMs from other fields may not generalize to medical oncology, a high-stakes clinical setting requiring rapid integration of new information. Objective: To evaluate the accuracy and safety of LLM answers on medical oncology examination questions. Design, Setting, and Participants: This cross-sectional study was conducted between May 28 and October 11, 2023. The American Society of Clinical Oncology (ASCO) Oncology Self-Assessment Series on ASCO Connection, the European Society of Medical Oncology (ESMO) Examination Trial questions, and an original set of board-style medical oncology multiple-choice questions were presented to 8 LLMs. Main Outcomes and Measures: The primary outcome was the percentage of correct answers. Medical oncologists evaluated the explanations provided by the best LLM for accuracy, classified the types of errors, and estimated the likelihood and extent of potential clinical harm. Results: Proprietary LLM 2 correctly answered 125 of 147 questions (85.0%; 95% CI, 78.2%-90.4%; P < .001 vs random answering). Proprietary LLM 2 outperformed an earlier version, proprietary LLM 1, which correctly answered 89 of 147 questions (60.5%; 95% CI, 52.2%-68.5%; P < .001), and the best open-source LLM, Mixtral-8x7B-v0.1, which correctly answered 87 of 147 questions (59.2%; 95% CI, 50.0%-66.4%; P < .001). The explanations provided by proprietary LLM 2 contained no or minor errors for 138 of 147 questions (93.9%; 95% CI, 88.7%-97.2%). Incorrect responses were most commonly associated with errors in information retrieval, particularly with recent publications, followed by erroneous reasoning and reading comprehension. If acted upon in clinical practice, 18 of 22 incorrect answers (81.8%; 95% CI, 59.7%-94.8%) would have a medium or high likelihood of moderate to severe harm. Conclusions and Relevance: In this cross-sectional study of the performance of LLMs on medical oncology examination questions, the best LLM answered questions with remarkable performance, although errors raised safety concerns. These results demonstrated an opportunity to develop and evaluate LLMs to improve health care clinician experiences and patient care, considering the potential impact on capabilities and safety.