Evaluating local open-source large language models for data extraction from unstructured reports on mechanical thrombectomy in patients with ischemic stroke.

BACKGROUND: A study was undertaken to assess the effectiveness of open-source large language models (LLMs) in extracting clinical data from unstructured mechanical thrombectomy reports in patients with ischemic stroke caused by a vessel occlusion. METHODS: We deployed local open-source LLMs to extract data points from free-text procedural reports in patients who underwent mechanical thrombectomy between September 2020 and June 2023 in our institution. The external dataset was obtained from a second university hospital and comprised consecutive cases treated between September 2023 and March 2024. Ground truth labeling was facilitated by a human-in-the-loop (HITL) approach, with time metrics recorded for both automated and manual data extractions. We tested three models-Mixtral, Qwen, and BioMistral-assessing their performance on precision, recall, and F1 score across 15 clinical categories such as National Institute of Health Stroke Scale (NIHSS) scores, occluded vessels, and medication details. RESULTS: The study included 1000 consecutive reports from our primary institution and 50 reports from a secondary institution. Mixtral showed the highest precision, achieving 0.99 for first series time extraction and 0.69 for occluded vessel identification within the internal dataset. In the external dataset, precision ranged from 1.00 for NIHSS scores to 0.70 for occluded vessels. Qwen showed moderate precision with a high of 0.85 for NIHSS scores and a low of 0.28 for occluded vessels. BioMistral had the broadest range of precision, from 0.81 for first series times to 0.14 for medication details. The HITL approach yielded an average time savings of 65.6% per case, with variations from 45.95% to 79.56%. CONCLUSION: This study highlights the potential of using LLMs for automated clinical data extraction from medical reports. Incorporating HITL annotations enhances precision and also ensures the reliability of the extracted data. This methodology presents a scalable privacy-preserving option that can significantly support clinical documentation and research endeavors.

External Validation and Retraining of DeepBleed: The First Open-Source 3D Deep Learning Network for the Segmentation of Spontaneous Intracerebral and Intraventricular Hemorrhage.

BACKGROUND: The objective of this study was to assess the performance of the first publicly available automated 3D segmentation for spontaneous intracerebral hemorrhage (ICH) based on a 3D neural network before and after retraining. METHODS: We performed an independent validation of this model using a multicenter retrospective cohort. Performance metrics were evaluated using the dice score (DSC), sensitivity, and positive predictive values (PPV). We retrained the original model (OM) and assessed the performance via an external validation design. A multivariate linear regression model was used to identify independent variables associated with the model's performance. Agreements in volumetric measurements and segmentation were evaluated using Pearson's correlation coefficients (r) and intraclass correlation coefficients (ICC), respectively. With 1040 patients, the OM had a median DSC, sensitivity, and PPV of 0.84, 0.79, and 0.93, compared to thoseo f 0.83, 0.80, and 0.91 in the retrained model (RM). However, the median DSC for infratentorial ICH was relatively low and improved significantly after retraining, at p < 0.001. ICH volume and location were significantly associated with the DSC, at p < 0.05. The agreement between volumetric measurements (r > 0.90, p > 0.05) and segmentations (ICC ≥ 0.9, p < 0.001) was excellent. CONCLUSION: The model demonstrated good generalization in an external validation cohort. Location-specific variances improved significantly after retraining. External validation and retraining are important steps to consider before applying deep learning models in new clinical settings.

Non-contrast computed tomography features predict intraventricular hemorrhage growth.

OBJECTIVES: Non-contrast computed tomography (NCCT) markers are robust predictors of parenchymal hematoma expansion in intracerebral hemorrhage (ICH). We investigated whether NCCT features can also identify ICH patients at risk of intraventricular hemorrhage (IVH) growth. METHODS: Patients with acute spontaneous ICH admitted at four tertiary centers in Germany and Italy were retrospectively included from January 2017 to June 2020. NCCT markers were rated by two investigators for heterogeneous density, hypodensity, black hole sign, swirl sign, blend sign, fluid level, island sign, satellite sign, and irregular shape. ICH and IVH volumes were semi-manually segmented. IVH growth was defined as IVH expansion > 1 mL (eIVH) or any delayed IVH (dIVH) on follow-up imaging. Predictors of eIVH and dIVH were explored with multivariable logistic regression. Hypothesized moderators and mediators were independently assessed in PROCESS macro models. RESULTS: A total of 731 patients were included, of whom 185 (25.31%) suffered from IVH growth, 130 (17.78%) had eIVH, and 55 (7.52%) had dIVH. Irregular shape was significantly associated with IVH growth (OR 1.68; 95%CI [1.16-2.44]; p = 0.006). In the subgroup analysis stratified by the IVH growth type, hypodensities were significantly associated with eIVH (OR 2.06; 95%CI [1.48-2.64]; p = 0.015), whereas irregular shape (OR 2.72; 95%CI [1.91-3.53]; p = 0.016) in dIVH. The association between NCCT markers and IVH growth was not mediated by parenchymal hematoma expansion. CONCLUSIONS: NCCT features identified ICH patients at a high risk of IVH growth. Our findings suggest the possibility to stratify the risk of IVH growth with baseline NCCT and might inform ongoing and future studies. CLINICAL RELEVANCE STATEMENT: Non-contrast CT features identified ICH patients at a high risk of intraventricular hemorrhage growth with subtype-specific differences. Our findings may assist in the risk stratification of intraventricular hemorrhage growth with baseline CT and might inform ongoing and future clinical studies. KEY POINTS: • NCCT features identified ICH patients at a high risk of IVH growth with subtype-specific differences. • The effect of NCCT features was not moderated by time and location or indirectly mediated by hematoma expansion. • Our findings may assist in the risk stratification of IVH growth with baseline NCCT and might inform ongoing and future studies.

Multilesion Segmentations in Patients with Intracerebral Hemorrhage: Reliability of ICH, IVH and PHE Masks.

Background and Purpose: Fully automated methods for segmentation and volume quantification of intraparenchymal hemorrhage (ICH), intraventricular hemorrhage extension (IVH), and perihematomal edema (PHE) are gaining increasing interest. Yet, reliabilities demonstrate considerable variances amongst each other. Our aim was therefore to evaluate both the intra- and interrater reliability of ICH, IVH and PHE on ground-truth segmentation masks. Methods: Patients with primary spontaneous ICH were retrospectively included from a German tertiary stroke center (Charité Berlin; January 2016−June 2020). Baseline and follow-up non-contrast Computed Tomography (NCCT) scans were analyzed for ICH, IVH, and PHE volume quantification by two radiology residents. Raters were blinded to all demographic and outcome data. Inter- and intrarater agreements were determined by calculating the Intraclass Correlation Coefficient (ICC) for a randomly selected set of patients with ICH, IVH, and PHE. Results: 100 out of 670 patients were included in the analysis. Interrater agreements ranged from an ICC of 0.998 for ICH (95% CI [0.993; 0.997]), to an ICC of 0.979 for IVH (95% CI [0.984; 0.993]), and an ICC of 0.886 for PHE (95% CI [0.760; 0.938]), all p-values < 0.001. Intrarater agreements ranged from an ICC of 0.997 for ICH (95% CI [0.996; 0.998]), to an ICC of 0.995 for IVH (95% CI [0.992; 0.996]), and an ICC of 0.980 for PHE (95% CI [0.971; 0.987]), all p-values < 0.001. Conclusion Manual segmentations of ICH, IVH, and PHE demonstrate good-to-excellent inter- and intrarater reliabilities, with the highest agreement for ICH and IVH and lowest for PHE. Therefore, the degree of variances reported in fully automated quantification methods might be related amongst others to variances in ground-truth masks.

Diagnostic Accuracy and Reliability of Noncontrast Computed Tomography Markers for Acute Hematoma Expansion among Radiologists.

BACKGROUND: Noncontrast Computed Tomography (NCCT) features are promising markers for acute hematoma expansion (HE) in patients with intracerebral hemorrhage (ICH). It remains unclear whether accurate identification of these markers is also reliable in raters with different levels of experience. METHODS: Patients with acute spontaneous ICH admitted at four tertiary centers in Germany and Italy were retrospectively included from January 2017 to June 2020. In total, nine NCCT markers were rated by one radiology resident, one radiology fellow, and one neuroradiology fellow with different levels experience in ICH imaging. Interrater reliabilities of the resident and radiology fellow were evaluated by calculated Cohen's kappa (κ) statistics in reference to the neuroradiology fellow who was referred as the gold standard. Gold-standard ratings were evaluated by calculated interrater κ statistics. Global interrater reliabilities were evaluated by calculated Fleiss kappa statistics across all three readers. A comparison of receiver operating characteristics (ROCs) was used to evaluate differences in the diagnostic accuracy for predicting acute hematoma expansion (HE) among the raters. RESULTS: Substantial-to-almost-perfect interrater concordance was found for the resident with interrater Cohen's kappa from 0.70 (95% CI 0.65-0.81) to 0.96 (95% CI 0.94-0.98). The interrater Cohen's kappa for the radiology fellow was moderate to almost perfect and ranged from 0.58 (95% CI 0.52-0.65) to 94 (95% CI 92-0.97). The intrarater gold-standard Cohen's kappa was almost perfect and ranged from 0.79 (95% CI 0.78-0.90) to 0.98 (95% CI 0.78-0.90). The global interrater Fleiss kappa ranged from 0.62 (95%CI 0.57-0.66) to 0.93 (95%CI 0.89-0.97). The diagnostic accuracy for the prediction of acute hematoma expansion (HE) was different for the island sign and fluid sign, with p-values < 0.05. CONCLUSION: The NCCT markers had a substantial-to-almost-perfect interrater agreement among raters with different levels of experience. Differences in the diagnostic accuracy for the prediction of acute HE were found in two out of nine NCCT markers. The study highlights the promising utility of NCCT markers for acute HE prediction.

Training with Odors Impacts Hippocampal Thickness in Patients with Mild Cognitive Impairment.

BACKGROUND: The olfactory system is affected early in Alzheimer's disease and olfactory loss can already be observed in patients with mild cognitive impairment (MCI). Olfactory training is effective for improving olfactory and cognitive function by stimulating the olfactory pathway, but its effect on patients with MCI remains unclear. OBJECTIVE: The aim of this randomized, prospective, controlled, blinded study was to assess whether a 4-month period of olfactory training (frequent short-term sniffing various odors) may have an effect on olfactory function, cognitive function, and morphology of medial temporal lobe (MTL) subregions and olfactory bulb in MCI patients. METHODS: A total of thirty-seven MCI patients were randomly assigned to the training group or a placebo group, which were performed twice a day for 4 months. Olfactory assessments, cognitive tests and magnetic resonance imaging were performed at the baseline and follow-up period. RESULTS: After the training, there was an increase in odor discrimination, and increased cortical thickness of bilateral hippocampus (CA23DG and CA1) and mean MTL. Additionally, the change of olfactory score was positively associated with change of volume of olfactory bulb and hippocampus; the change of global cognition was positively associated with change of cortical thickness of hippocampus, entorhinal cortex and mean MTL; the change of cortical thickness of entorhinal cortex was positively associated with change of executive function. CONCLUSION: Olfactory training was associated with an increase in cortical thickness of the hippocampus but not olfactory bulb volume in patients with MCI. Olfactory training may serve as an early intervention of preventing hippocampal atrophy.

Automatic Segmentation of the Olfactory Bulb.

The olfactory bulb (OB) has an essential role in the human olfactory pathway. A change in olfactory function is associated with a change of OB volume. It has been shown to predict the prognosis of olfactory loss and its volume is a biomarker for various neurodegenerative diseases, such as Alzheimer's disease. Thus far, obtaining an OB volume for research purposes has been performed by manual segmentation alone; a very time-consuming and highly rater-biased process. As such, this process dramatically reduces the ability to produce fair and reliable comparisons between studies, as well as the processing of large datasets. Our study aims to solve this by proposing a novel methodological framework for the unbiased measurement of OB volume. In this paper, we present a fully automated tool that successfully performs such a task, accurately and quickly. In order to develop a stable and versatile algorithm and to train the neural network, we used four datasets consisting of whole-brain T1 and high-resolution T2 MRI scans, as well as the corresponding clinical information of the subject's smelling ability. One dataset contained data of patients suffering from anosmia or hyposmia (N = 79), and the other three datasets contained data of healthy controls (N = 91). First, the manual segmentation labels of the OBs were created by two experienced raters, independently and blinded. The algorithm consisted of the following four different steps: (1) multimodal data co-registration of whole-brain T1 images and T2 images, (2) template-based localization of OBs, (3) bounding box construction, and lastly, (4) segmentation of the OB using a 3D-U-Net. The results from the automated segmentation algorithm were tested on previously unseen data, achieving a mean dice coefficient (DC) of 0.77 ± 0.05, which is remarkably convergent with the inter-rater DC of 0.79 ± 0.08 estimated for the same cohort. Additionally, the symmetric surface distance (ASSD) was 0.43 ± 0.10. Furthermore, the segmentations produced using our algorithm were manually rated by an independent blinded rater and have reached an equivalent rating score of 5.95 ± 0.87 compared to a rating score of 6.23 ± 0.87 for the first rater's segmentation and 5.92 ± 0.81 for the second rater's manual segmentation. Taken together, these results support the success of our tool in producing automatic fast (3-5 min per subject) and reliable segmentations of the OB, with virtually matching accuracy with the current gold standard technique for OB segmentation. In conclusion, we present a newly developed ready-to-use tool that can perform the segmentation of OBs based on multimodal data consisting of T1 whole-brain images and T2 coronal high-resolution images. The accuracy of the segmentations predicted by the algorithm matches the manual segmentations made by two well-experienced raters. This method holds potential for immediate implementation in clinical practice. Furthermore, its ability to perform quick and accurate processing of large datasets may provide a valuable contribution to advancing our knowledge of the olfactory system, in health and disease. Specifically, our framework may integrate the use of olfactory bulb volume (OBV) measurements for the diagnosis and treatment of olfactory loss and improve the prognosis and treatment options of olfactory dysfunctions.