A Comparison Between GPT-3.5, GPT-4, and GPT-4V: Can the Large Language Model (ChatGPT) Pass the Japanese Board of Orthopaedic Surgery Examination?

Introduction Recently, large-scale language models, such as ChatGPT (OpenAI, San Francisco, CA), have evolved. These models are designed to think and act like humans and possess a broad range of specialized knowledge. GPT-3.5 was reported to be at a level of passing the United States Medical Licensing Examination. Its capabilities continue to evolve, and in October 2023, GPT-4V became available as a model capable of image recognition. Therefore, it is important to know the current performance of these models because they will be soon incorporated into medical practice. We aimed to evaluate the performance of ChatGPT in the field of orthopedic surgery. Methods We used three years' worth of Japanese Board of Orthopaedic Surgery Examinations (JBOSE) conducted in 2021, 2022, and 2023. Questions and their multiple-choice answers were used in their original Japanese form, as was the official examination rubric. We inputted these questions into three versions of ChatGPT: GPT-3.5, GPT-4, and GPT-4V. For image-based questions, we inputted only textual statements for GPT-3.5 and GPT-4, and both image and textual statements for GPT-4V. As the minimum scoring rate acquired to pass is not officially disclosed, it was calculated using publicly available data. Results The estimated minimum scoring rate acquired to pass was calculated as 50.1% (43.7-53.8%). For GPT-4, even when answering all questions, including the image-based ones, the percentage of correct answers was 59% (55-61%) and GPT-4 was able to achieve the passing line. When excluding image-based questions, the score reached 67% (63-73%). For GPT-3.5, the percentage was limited to 30% (28-32%), and this version could not pass the examination. There was a significant difference in the performance between GPT-4 and GPT-3.5 (p < 0.001). For image-based questions, the percentage of correct answers was 25% in GPT-3.5, 38% in GPT-4, and 38% in GPT-4V. There was no significant difference in the performance for image-based questions between GPT-4 and GPT-4V. Conclusions ChatGPT had enough performance to pass the orthopedic specialist examination. After adding further training data such as images, ChatGPT is expected to be applied to the orthopedics field.

Bimodal artificial intelligence using TabNet for differentiating spinal cord tumors-Integration of patient background information and images.

We proposed a bimodal artificial intelligence that integrates patient information with images to diagnose spinal cord tumors. Our model combines TabNet, a state-of-the-art deep learning model for tabular data for patient information, and a convolutional neural network for images. As training data, we collected 259 spinal tumor patients (158 for schwannoma and 101 for meningioma). We compared the performance of the image-only unimodal model, table-only unimodal model, bimodal model using a gradient-boosting decision tree, and bimodal model using TabNet. Our proposed bimodal model using TabNet performed best (area under the receiver-operating characteristic curve [AUROC]: 0.91) in the training data and significantly outperformed the physicians' performance. In the external validation using 62 cases from the other two facilities, our bimodal model showed an AUROC of 0.92, proving the robustness of the model. The bimodal analysis using TabNet was effective for differentiating spinal tumors.

Development of artificial intelligence for automated measurement of cervical lordosis on lateral radiographs.

Cervical sagittal alignment is an essential parameter for the evaluation of spine disorders. Manual measurement is time-consuming and burdensome to measurers. Artificial intelligence (AI) in the form of convolutional neural networks has begun to be used to measure x-rays. This study aimed to develop AI for automated measurement of lordosis on lateral cervical x-rays. We included 4546 cervical x-rays from 1674 patients. For all x-rays, the caudal endplates of C2 and C7 were labeled based on consensus among well-experienced spine surgeons, the data for which were used as ground truth. This ground truth was split into training data and test data, and the AI model learned the training data. The absolute error of the AI measurements relative to the ground truth for 4546 x-rays was determined by fivefold cross-validation. Additionally, the absolute error of AI measurements was compared with the error of other 2 surgeons' measurements on 415 radiographs of 168 randomly selected patients. In fivefold cross-validation, the absolute error of the AI model was 3.3° in the average and 2.2° in the median. For comparison of other surgeons, the mean absolute error for measurement of 168 patients was 3.1° ± 3.4° for the AI model, 3.9° ± 3.4° for Surgeon 1, and 3.8° ± 4.7° for Surgeon 2. The AI model had a significantly smaller error than Surgeon 1 and Surgeon 2 (P = 0.002 and 0.036). This algorithm is available at ( https://ykszk.github.io/c2c7demo/ ). The AI model measured cervical spine alignment with better accuracy than surgeons. AI can assist in routine medical care and can be helpful in research that measures large numbers of images. However, because of the large errors in rare cases such as highly deformed ones, AI may, in principle, be limited to assisting humans.

Epidemiology of symptomatic ossification of the posterior longitudinal ligament: a nationwide registry survey.

BACKGROUND: Ossification of the posterior longitudinal ligament (OPLL) is radiographically detectable in 3-6% of Asian individuals, although not all detectable OPLL cases lead to myelopathy. To date, it is unknown how many patients suffer from neurological symptoms due to OPLL. The purpose of this study was to investigate the epidemiology of symptomatic OPLL using Japan's national registry database. METHODS: We examined the registry data of patients with OPLL who held a certificate of medical subsidy from the Japanese Ministry of Health, Labor and Welfare. The study period was from January 1, 2011 to December 31, 2012. RESULTS: Registry data revealed that the incidence and the period prevalence of symptomatic OPLL were 0.005% (5 per 100,000 population) and 0.027% (27 per 100,000 population), respectively. OPLL occurred twice as often in men as in women. The peak age for onset of symptoms was 60-69 years. The mean Japanese Orthopedic Association (JOA) score was 9 points. Ninety percent of OPLL patients underwent surgery, and 90% of these surgeries were performed with a posterior approach. The most common indication for surgery was a JOA score of 11 points. CONCLUSIONS: According to registry data, the prevalence of symptomatic OPLL was less than one-hundredth of that of radiographically detected OPLL. This indicates that most cases of radiographically detectable OPLL may be asymptomatic.