Assessing the Proficiency of LLMs with Various Tasks and Evaluators.

Previous studies have been limited to giving one or two tasks to Large Language Models (LLMs) and involved a small number of evaluators within a single domain to evaluate the LLM's answer. We assessed the proficiency of four LLMs by applying eight tasks and evaluating 32 results with 17 evaluators from diverse domains, demonstrating the significance of various tasks and evaluators on LLMs.

Fill in the Blank, MIMIC the Procedure: Tracking Patient Record for Medical Data Reconstruction.

This study addresses the missing data problem in the large-scale medical dataset MIMIC-IV, especially in situations where intubation-extubation events are paired. We employed a strategy involving patient scenario works that checked the temporal order and logical links of intubation/extubation data, and seven reconstruction rules for handling missing values. Through this, we reduced the overall loss rate from 36.89% (3321 records) to 13.37% (1204 records) and achieved a 37.26% data increase (+2117 records) compared to before reconstruction(6582).

Well-Tempered Medical Prompt Engineering for Explainable Extubation.

This study investigated whether the large language model (LLM) utilizes sufficient domain knowledge to reason about critical medical events such as extubation. In detail, we tested whether the LLM accurately comprehends given tabular data and variable importance and whether it can be used in complement to existing ML models such as XGBoost.

Automation of Trainable Datasets Generation for Medical-Specific Language Model: Using MIMIC-IV Discharge Notes.

This study introduces a novel approach for generating machine-generated instruction datasets for fine-tuning medical-specialized language models using MIMIC-IV discharge records. The study created a large-scale text dataset comprising instructions, cropped discharge notes as inputs, and outputs in JSONL format. The dataset was generated through three main stages, generating instruction and output using seed tasks provided by medical experts, followed by invalid data filtering. The generated dataset consisted of 51,385 sets, with mean ROUGE between seed tasks of 0.185. Evaluation of the generated dataset were promising, with high validity rates determined by both GPT-3.5 and a human annotator (88.0% and 88.5% respectively). The study highlights the potential of automating dataset creation for NLP tasks in the medical domain.

Machine Learning-Based Predictive Models for Early Detection of Cardiovascular Diseases: A Study Utilizing Patient Samples from a Tertiary Health Promotion Center in Korea.

A machine learning model was developed for cardiovascular diseases prediction based on 21,118 patient checkups data from a tertiary medical institution in Seoul, Korea, collected between 2009 and 2021. XGBoost algorithm showed the highest predictive performance, with an average AUROC of 0.877. In survival analysis, XGBSE achieved an AUROC exceeding 0.9 for 2-9 year predictions, with a C-index of 0.878 across all diseases, outperforming Cox regression (C-index of 0.887). A high-performance prediction model for cardiovascular diseases using the XGBSE algorithm was successfully developed and is poised for real-world clinical application following external simplification and validation.

Comparison of AI with and without hand-crafted features to classify Alzheimer's disease in different languages.

BACKGROUND: Detecting and analyzing Alzheimer's disease (AD) in its early stages is a crucial and significant challenge. Speech data from AD patients can aid in diagnosing AD since the speech features have common patterns independent of race and spoken language. However, previous models for diagnosing AD from speech data have often focused on the characteristics of a single language, with no guarantee of scalability to other languages. In this study, we used the same method to extract acoustic features from two language datasets to diagnose AD. METHODS: Using the Korean and English speech datasets, we used ten models capable of real-time AD and healthy control classification, regardless of language type. Four machine learning models were based on hand-crafted features, while the remaining six deep learning models utilized non-explainable features. RESULTS: The highest accuracy achieved by the machine learning models was 0.73 and 0.69 for the Korean and English speech datasets, respectively. The deep learning models' maximum achievable accuracy reached 0.75 and 0.78, with their minimum classification time of 0.01s and 0.02s. These findings reveal the models' robustness regardless of Korean and English and real-time diagnosis of AD through a 30-s voice sample. CONCLUSION: Non-explainable deep learning models that directly acquire voice representations surpassed machine learning models utilizing hand-crafted features in AD diagnosis. In addition, these AI models could confirm the possibility of extending to a language-agnostic AD diagnosis.

Improved prediction of clinical pregnancy using artificial intelligence with enhanced inner cell mass and trophectoderm images.

This study aimed to assess the performance of an artificial intelligence (AI) model for predicting clinical pregnancy using enhanced inner cell mass (ICM) and trophectoderm (TE) images. In this retrospective study, we included static images of 2555 day-5-blastocysts from seven in vitro fertilization centers in South Korea. The main outcome of the study was the predictive capability of the model to detect clinical pregnancies (gestational sac). Compared with the original embryo images, the use of enhanced ICM and TE images improved the average area under the receiver operating characteristic curve for the AI model from 0.716 to 0.741. Additionally, a gradient-weighted class activation mapping analysis demonstrated that the enhanced image-trained AI model was able to extract features from crucial areas of the embryo in 99% (506/512) of the cases. Particularly, it could extract the ICM and TE. In contrast, the AI model trained on the original images focused on the main areas in only 86% (438/512) of the cases. Our results highlight the potential efficacy of using ICM- and TE-enhanced embryo images when training AI models to predict clinical pregnancy.

Prediction of Cardiovascular Complication in Patients with Newly Diagnosed Type 2 Diabetes Using an XGBoost/GRU-ODE-Bayes-Based Machine-Learning Algorithm.

BACKGRUOUND: Cardiovascular disease is life-threatening yet preventable for patients with type 2 diabetes mellitus (T2DM). Because each patient with T2DM has a different risk of developing cardiovascular complications, the accurate stratification of cardiovascular risk is critical. In this study, we proposed cardiovascular risk engines based on machine-learning algorithms for newly diagnosed T2DM patients in Korea. METHODS: To develop the machine-learning-based cardiovascular disease engines, we retrospectively analyzed 26,166 newly diagnosed T2DM patients who visited Seoul St. Mary's Hospital between July 2009 and April 2019. To accurately measure diabetes-related cardiovascular events, we designed a buffer (1 year), an observation (1 year), and an outcome period (5 years). The entire dataset was split into training and testing sets in an 8:2 ratio, and this procedure was repeated 100 times. The area under the receiver operating characteristic curve (AUROC) was calculated by 10-fold cross-validation on the training dataset. RESULTS: The machine-learning-based risk engines (AUROC XGBoost=0.781±0.014 and AUROC gated recurrent unit [GRU]-ordinary differential equation [ODE]-Bayes=0.812±0.016) outperformed the conventional regression-based model (AUROC=0.723± 0.036). CONCLUSION: GRU-ODE-Bayes-based cardiovascular risk engine is highly accurate, easily applicable, and can provide valuable information for the individualized treatment of Korean patients with newly diagnosed T2DM.

Development and Validation of a Convolutional Neural Network Model to Predict a Pathologic Fracture in the Proximal Femur Using Abdomen and Pelvis CT Images of Patients With Advanced Cancer.

BACKGROUND: Improvement in survival in patients with advanced cancer is accompanied by an increased probability of bone metastasis and related pathologic fractures (especially in the proximal femur). The few systems proposed and used to diagnose impending fractures owing to metastasis and to ultimately prevent future fractures have practical limitations; thus, novel screening tools are essential. A CT scan of the abdomen and pelvis is a standard modality for staging and follow-up in patients with cancer, and radiologic assessments of the proximal femur are possible with CT-based digitally reconstructed radiographs. Deep-learning models, such as convolutional neural networks (CNNs), may be able to predict pathologic fractures from digitally reconstructed radiographs, but to our knowledge, they have not been tested for this application. QUESTIONS/PURPOSES: (1) How accurate is a CNN model for predicting a pathologic fracture in a proximal femur with metastasis using digitally reconstructed radiographs of the abdomen and pelvis CT images in patients with advanced cancer? (2) Do CNN models perform better than clinicians with varying backgrounds and experience levels in predicting a pathologic fracture on abdomen and pelvis CT images without any knowledge of the patients' histories, except for metastasis in the proximal femur? METHODS: A total of 392 patients received radiation treatment of the proximal femur at three hospitals from January 2011 to December 2021. The patients had 2945 CT scans of the abdomen and pelvis for systemic evaluation and follow-up in relation to their primary cancer. In 33% of the CT scans (974), it was impossible to identify whether a pathologic fracture developed within 3 months after each CT image was acquired, and these were excluded. Finally, 1971 cases with a mean age of 59 ± 12 years were included in this study. Pathologic fractures developed within 3 months after CT in 3% (60 of 1971) of cases. A total of 47% (936 of 1971) were women. Sixty cases had an established pathologic fracture within 3 months after each CT scan, and another group of 1911 cases had no established pathologic fracture within 3 months after CT scan. The mean age of the cases in the former and latter groups was 64 ± 11 years and 59 ± 12 years, respectively, and 32% (19 of 60) and 53% (1016 of 1911) of cases, respectively, were female. Digitally reconstructed radiographs were generated with perspective projections of three-dimensional CT volumes onto two-dimensional planes. Then, 1557 images from one hospital were used for a training set. To verify that the deep-learning models could consistently operate even in hospitals with a different medical environment, 414 images from other hospitals were used for external validation. The number of images in the groups with and without a pathologic fracture within 3 months after each CT scan increased from 1911 to 22,932 and from 60 to 720, respectively, using data augmentation methods that are known to be an effective way to boost the performance of deep-learning models. Three CNNs (VGG16, ResNet50, and DenseNet121) were fine-tuned using digitally reconstructed radiographs. For performance measures, the area under the receiver operating characteristic curve, accuracy, sensitivity, specificity, precision, and F1 score were determined. The area under the receiver operating characteristic curve was used to evaluate three CNN models mainly, and the optimal accuracy, sensitivity, and specificity were calculated using the Youden J statistic. Accuracy refers to the proportion of fractures in the groups with and without a pathologic fracture within 3 months after each CT scan that were accurately predicted by the CNN model. Sensitivity and specificity represent the proportion of accurately predicted fractures among those with and without a pathologic fracture within 3 months after each CT scan, respectively. Precision is a measure of how few false-positives the model produces. The F1 score is a harmonic mean of sensitivity and precision, which have a tradeoff relationship. Gradient-weighted class activation mapping images were created to check whether the CNN model correctly focused on potential pathologic fracture regions. The CNN model with the best performance was compared with the performance of clinicians. RESULTS: DenseNet121 showed the best performance in identifying pathologic fractures; the area under the receiver operating characteristic curve for DenseNet121 was larger than those for VGG16 (0.77 ± 0.07 [95% CI 0.75 to 0.79] versus 0.71 ± 0.08 [95% CI 0.69 to 0.73]; p = 0.001) and ResNet50 (0.77 ± 0.07 [95% CI 0.75 to 0.79] versus 0.72 ± 0.09 [95% CI 0.69 to 0.74]; p = 0.001). Specifically, DenseNet121 scored the highest in sensitivity (0.22 ± 0.07 [95% CI 0.20 to 0.24]), precision (0.72 ± 0.19 [95% CI 0.67 to 0.77]), and F1 score (0.34 ± 0.10 [95% CI 0.31 to 0.37]), and it focused accurately on the region with the expected pathologic fracture. Further, DenseNet121 was less likely than clinicians to mispredict cases in which there was no pathologic fracture than cases in which there was a fracture; the performance of DenseNet121 was better than clinician performance in terms of specificity (0.98 ± 0.01 [95% CI 0.98 to 0.99] versus 0.86 ± 0.09 [95% CI 0.81 to 0.91]; p = 0.01), precision (0.72 ± 0.19 [95% CI 0.67 to 0.77] versus 0.11 ± 0.10 [95% CI 0.05 to 0.17]; p = 0.0001), and F1 score (0.34 ± 0.10 [95% CI 0.31 to 0.37] versus 0.17 ± 0.15 [95% CI 0.08 to 0.26]; p = 0.0001). CONCLUSION: CNN models may be able to accurately predict impending pathologic fractures from digitally reconstructed radiographs of the abdomen and pelvis CT images that clinicians may not anticipate; this can assist medical, radiation, and orthopaedic oncologists clinically. To achieve better performance, ensemble-learning models using knowledge of the patients' histories should be developed and validated. The code for our model is publicly available online at https://github.com/taehoonko/CNN_path_fx_prediction . LEVEL OF EVIDENCE: Level III, diagnostic study.

Improving Machine Learning Diabetes Prediction Models for the Utmost Clinical Effectiveness.

The early prediction of diabetes can facilitate interventions to prevent or delay it. This study proposes a diabetes prediction model based on machine learning (ML) to encourage individuals at risk of diabetes to employ healthy interventions. A total of 38,379 subjects were included. We trained the model on 80% of the subjects and verified its predictive performance on the remaining 20%. Furthermore, the performances of several algorithms were compared, including logistic regression, decision tree, random forest, eXtreme Gradient Boosting (XGBoost), Cox regression, and XGBoost Survival Embedding (XGBSE). The area under the receiver operating characteristic curve (AUROC) of the XGBoost model was the largest, followed by those of the decision tree, logistic regression, and random forest models. For the survival analysis, XGBSE yielded an AUROC exceeding 0.9 for the 2- to 9-year predictions and a C-index of 0.934, while the Cox regression achieved a C-index of 0.921. After lowering the threshold from 0.5 to 0.25, the sensitivity increased from 0.011 to 0.236 for the 2-year prediction model and from 0.607 to 0.994 for the 9-year prediction model, while the specificity showed negligible changes. We developed a high-performance diabetes prediction model that applied the XGBSE algorithm with threshold adjustment. We plan to use this prediction model in real clinical practice for diabetes prevention after simplifying and validating it externally.

Deep Learning Model Based on 3D Optical Coherence Tomography Images for the Automated Detection of Pathologic Myopia.

Pathologic myopia causes vision impairment and blindness, and therefore, necessitates a prompt diagnosis. However, there is no standardized definition of pathologic myopia, and its interpretation by 3D optical coherence tomography images is subjective, requiring considerable time and money. Therefore, there is a need for a diagnostic tool that can automatically and quickly diagnose pathologic myopia in patients. This study aimed to develop an algorithm that uses 3D optical coherence tomography volumetric images (C-scan) to automatically diagnose patients with pathologic myopia. The study was conducted using 367 eyes of patients who underwent optical coherence tomography tests at the Ophthalmology Department of Incheon St. Mary's Hospital and Seoul St. Mary's Hospital from January 2012 to May 2020. To automatically diagnose pathologic myopia, a deep learning model was developed using 3D optical coherence tomography images. The model was developed using transfer learning based on four pre-trained convolutional neural networks (ResNet18, ResNext50, EfficientNetB0, EfficientNetB4). Grad-CAM was used to visualize features affecting the detection of pathologic myopia. The performance of each model was evaluated and compared based on accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC). The model based on EfficientNetB4 showed the best performance (95% accuracy, 93% sensitivity, 96% specificity, and 98% AUROC) in identifying pathologic myopia.

Estimation of low-density lipoprotein cholesterol levels using machine learning.

BACKGROUND: Low-density lipoprotein-cholesterol (LDL-C) is used as a threshold and target for treating dyslipidemia. Although the Friedewald equation is widely used to estimate LDL-C, it has been known to be inaccurate in the case of high triglycerides (TG) or non-fasting states. We aimed to propose a novel method to estimate LDL-C using machine learning. METHODS: Using a large, single-center electronic health record database, we derived a ML algorithm to estimate LDL-C from standard lipid profiles. From 1,029,572 cases with both standard lipid profiles (total cholesterol, high-density lipoprotein-cholesterol, and TG) and direct LDL-C measurements, 823,657 tests were used to derive LDL-C estimation models. Patient characteristics such as sex, age, height, weight, and other laboratory values were additionally used to create separate data sets and algorithms. RESULTS: Machine learning with gradient boosting (LDL-CX) and neural network (LDL-CN) showed better correlation with directly measured LDL-C, compared with conventional methods (r = 0.9662, 0.9668, 0.9563, 0.9585; for LDL-CX, LDL-CN, Friedewald [LDL-CF], and Martin [LDL-CM] equations, respectively). The overall bias of LDL-CX (-0.27 mg/dL, 95% CI -0.30 to -0.23) and LDL-CN (-0.01 mg/dL, 95% CI -0.04-0.03) were significantly smaller compared with both LDL-CF (-3.80 mg/dL, 95% CI -3.80 to -3.60) or LDL-CM (-2.00 mg/dL, 95% CI -2.00 to -1.94), especially at high TG levels. CONCLUSIONS: Machine learning algorithms were superior in estimating LDL-C compared with the conventional Friedewald or the more contemporary Martin equations. Through external validation and modification, machine learning could be incorporated into electronic health records to substitute LDL-C estimation.

Asbestosis diagnosis algorithm combining the lung segmentation method and deep learning model in computed tomography image.

BACKGROUND: Early detection of asbestosis is important; hence, quick and accurate diagnostic tools are essential. This study aimed to develop an algorithm that combines lung segmentation and deep learning models that can be utilized as a clinical decision support system (CDSS) for diagnosing patients with asbestosis in segmented computed tomography (CT) images. METHODS: We accurately segmented the lungs in CT images of patients examined at Seoul St. Mary's Hospital using a threshold-based method. Lungs with asbestosis and normal lungs were classified by applying the segmented image to the long-term recurrent convolutional network deep learning model. Performance was evaluated using the area under the receiver operating characteristic curve (AUROC) and F1 score from the test data. RESULTS: The algorithm developed using the DenseNet201pre-trained model showed excellent performance, with a sensitivity of 0.962, specificity of 0.975, accuracy of 0.970, AUROC of 0.968, and F1 score of 0.961. CONCLUSIONS: We developed an algorithm with significantly better diagnostic accuracy than a radiologist (0.970 vs. 0.73-0.79). Our developed algorithm is expected to be an excellent support tool if used as a CDSS to diagnose asbestosis using CT images.

Preparation of Network-Structured Carbon Nanofiber Mats Based on PAN Blends Using Electrospinning and Hot-Pressing Methods for Supercapacitor Applications.

In this work, we prepared network-structured carbon nanofibers using polyacrylonitrile blends (PAN150 and PAN85) with different molecular weights (150,000 and 85,000 g mol-1) as precursors through electrospinning/hot-pressing methods and stabilization/carbonization processes. The obtained PAN150/PAN85 polymer nanofibers (PNFs; PNF-73, PNF-64 and PNF-55) with different weight ratios of 70/30, 60/40 and 50/50 (w/w) provided good mechanical and electrochemical properties due to the formation of physically bonded network structures between the blended PAN nanofibers during the hot-processing/stabilization processes. The resulting carbonized PNFs (cPNFs; cPNF-73, cPNF-64, and cPNF-55) were utilized as anode materials for supercapacitor applications. cPNF-73 exhibited a good specific capacitance of 689 F g-1 at 1 A g-1 in a three-electrode set-up compared to cPNF-64 (588 F g-1 at 1 A g-1) and cPNF-55 (343 F g-1 at 1 A g-1). In addition, an asymmetric hybrid cPNF-73//NiCo2O4 supercapacitor device also showed a good specific capacitance of 428 F g-1 at 1 A g-1 compared to cPNF-64 (400 F g-1 at 1 A g-1) and cPNF-55 (315 F g-1 at 1 A g-1). The cPNF-73-based device showed a good energy density of 1.74 W h kg-1 (0.38 W kg-1) as well as an excellent cyclic stability (83%) even after 2000 continuous charge-discharge cycles at a current density of 2 A g-1.

Markers of Myocardial Damage Predict Mortality in Patients With Aortic Stenosis.

BACKGROUND: Cardiovascular magnetic resonance (CMR) is increasingly used for risk stratification in aortic stenosis (AS). However, the relative prognostic power of CMR markers and their respective thresholds remains undefined. OBJECTIVES: Using machine learning, the study aimed to identify prognostically important CMR markers in AS and their thresholds of mortality. METHODS: Patients with severe AS undergoing AVR (n = 440, derivation; n = 359, validation cohort) were prospectively enrolled across 13 international sites (median 3.8 years' follow-up). CMR was performed shortly before surgical or transcatheter AVR. A random survival forest model was built using 29 variables (13 CMR) with post-AVR death as the outcome. RESULTS: There were 52 deaths in the derivation cohort and 51 deaths in the validation cohort. The 4 most predictive CMR markers were extracellular volume fraction, late gadolinium enhancement, indexed left ventricular end-diastolic volume (LVEDVi), and right ventricular ejection fraction. Across the whole cohort and in asymptomatic patients, risk-adjusted predicted mortality increased strongly once extracellular volume fraction exceeded 27%, while late gadolinium enhancement >2% showed persistent high risk. Increased mortality was also observed with both large (LVEDVi >80 mL/m2) and small (LVEDVi ≤55 mL/m2) ventricles, and with high (>80%) and low (≤50%) right ventricular ejection fraction. The predictability was improved when these 4 markers were added to clinical factors (3-year C-index: 0.778 vs 0.739). The prognostic thresholds and risk stratification by CMR variables were reproduced in the validation cohort. CONCLUSIONS: Machine learning identified myocardial fibrosis and biventricular remodeling markers as the top predictors of survival in AS and highlighted their nonlinear association with mortality. These markers may have potential in optimizing the decision of AVR.

Satisfaction and Usability of an Information and Communications Technology-Based System by Clinically Healthy Patients With COVID-19 and Medical Professionals: Cross-sectional Survey and Focus Group Interview Study.

BACKGROUND: Digital health care is an important strategy in the war against COVID-19. South Korea introduced living and treatment support centers (LTSCs) to control regional outbreaks and care for patients with asymptomatic or mild COVID-19. Seoul National University Hospital (SNUH) introduced information and communications technology (ICT)-based solutions to manage clinically healthy patients with COVID-19. OBJECTIVE: This study aims to investigate satisfaction and usability by patients and health professionals in the optimal use of a mobile app and wearable device that SNUH introduced to the LTSC for clinically healthy patients with COVID-19. METHODS: Online surveys and focus group interviews were conducted to collect quantitative and qualitative data. RESULTS: Regarding usability testing of the wearable device, perceived usefulness had the highest mean score of 4.45 (SD 0.57) points out of 5. Regarding usability of the mobile app, perceived usefulness had the highest mean score of 4.62 (SD 0.48) points out of 5. Regarding satisfaction items for the mobile app among medical professionals, the "self-reporting" item had the highest mean score of 4.42 (SD 0.58) points out of 5. In focus group interviews of health care professionals, hospital information system interfacing was the most important functional requirement for ICT-based COVID-19 telemedicine. CONCLUSIONS: Improvement of patient safety and reduction of the burden on medical staff were the expected positive outcomes. Stability and reliability of the device, patient education, accountability, and reimbursement issues should be considered as part of the development of remote patient monitoring. In responding to a novel contagious disease, telemedicine and a wearable device were shown to be useful during a global crisis.

Privacy-Oriented Technique for COVID-19 Contact Tracing (PROTECT) Using Homomorphic Encryption: Design and Development Study.

BACKGROUND: Various techniques are used to support contact tracing, which has been shown to be highly effective against the COVID-19 pandemic. To apply the technology, either quarantine authorities should provide the location history of patients with COVID-19, or all users should provide their own location history. This inevitably exposes either the patient's location history or the personal location history of other users. Thus, a privacy issue arises where the public good (via information release) comes in conflict with privacy exposure risks. OBJECTIVE: The objective of this study is to develop an effective contact tracing system that does not expose the location information of the patient with COVID-19 to other users of the system, or the location information of the users to the quarantine authorities. METHODS: We propose a new protocol called PRivacy Oriented Technique for Epidemic Contact Tracing (PROTECT) that securely shares location information of patients with users by using the Brakerski/Fan-Vercauteren homomorphic encryption scheme, along with a new, secure proximity computation method. RESULTS: We developed a mobile app for the end-user and a web service for the quarantine authorities by applying the proposed method, and we verified their effectiveness. The proposed app and web service compute the existence of intersections between the encrypted location history of patients with COVID-19 released by the quarantine authorities and that of the user saved on the user's local device. We also found that this contact tracing smartphone app can identify whether the user has been in contact with such patients within a reasonable time. CONCLUSIONS: This newly developed method for contact tracing shares location information by using homomorphic encryption, without exposing the location information of patients with COVID-19 and other users. Homomorphic encryption is challenging to apply to practical issues despite its high security value. In this study, however, we have designed a system using the Brakerski/Fan-Vercauteren scheme that is applicable to a reasonable size and developed it to an operable format. The developed app and web service can help contact tracing for not only the COVID-19 pandemic but also other epidemics.