Data Quality Estimation Via Model Performance: Machine Learning as a Validation Tool.

In our recent study, the attempt to classify neurosurgical operative reports into routinely used expert-derived classes exhibited an F-score not exceeding 0.74. This study aimed to test how improving the classifier (target variable) affected the short text classification with deep learning on real-world data. We redesigned the target variable based on three strict principles when applicable: pathology, localization, and manipulation type. The deep learning significantly improved with the best result of operative report classification into 13 classes (accuracy = 0.995, F1 = 0.990). Reasonable text classification with machine learning should be a two-way process: the model performance must be ensured by the unambiguous textual representation reflected in corresponding target variables. At the same time, the validity of human-generated codification can be inspected via machine learning.

Multinomial Classification of Neurosurgical Operations Using Gradient Boosting and Deep Learning Algorithms.

This study aimed at testing the feasibility of neurosurgical procedures classification into 100+ classes using natural language processing and machine learning. A catboost algorithm and bidirectional recurrent neural network with a gated recurrent unit showed almost the same accuracy of ∼81%, with suggestions of correct class in top 2-3 scored classes up to 98.9%. The classification of neurosurgical procedures via machine learning appears to be a technically solvable task which can be additionally improved considering data enhancement and classes verification.

Predicting the Length of Stay in Neurosurgery with RuGPT-3 Language Model.

In this study, we update the evaluation of the Russian GPT3 model presented in our previous paper in predicting the length of stay (LOS) in neurosurgery. We aimed to assess the performance the Russian GPT-3 (ruGPT-3) language model in LOS prediction using narrative medical records in neurosurgery compared to doctors' and patients' expectations. Doctors appeared to have the most realistic LOS expectations (MAE = 2.54), while the model's predictions (MAE = 3.53) were closest to the patients' (MAE = 3.47) but inferior to them (p = 0.011). A detailed analysis showed a solid quality of ruGPT-3 performance based on narrative clinical texts. Considering our previous findings obtained with recurrent neural networks and FastText vector representation, we estimate the new result as important but probably improveable.

The Classification of Scientific Abstracts Using Text Statistical Features.

Automated abstracts classification could significantly facilitate scientific literature screening. The classification of short texts could be based on their statistical properties. This research aimed to evaluate the quality of short medical abstracts classification primarily based on text statistical features. Twelve experiments with machine learning models over the sets of text features were performed on a dataset of 671 article abstracts. Each experiment was repeated 300 times to estimate the classification quality, ending up with 3600 tests total. We achieved the best result (F1 = 0.775) using a random forest machine learning model with keywords and three-dimensional Word2Vec embeddings. The classification of scientific abstracts might be implemented using straightforward and computationally inexpensive methods presented in this paper. The approach we described is expected to facilitate literature selection by researchers.

A Comparison of Word Embeddings to Study Complications in Neurosurgery.

Our study aimed to compare the capability of different word embeddings to capture the semantic similarity of clinical concepts related to complications in neurosurgery at the level of medical experts. Eighty-four sets of word embeddings (based on Word2vec, GloVe, FastText, PMI, and BERT algorithms) were benchmarked in a clustering task. FastText model showed the best close to the medical expertise capability to group medical terms by their meaning (adjusted Rand index = 0.682). Word embedding models can accurately reflect clinical concepts' semantic and linguistic similarities, promising their robust usage in medical domain-specific NLP tasks.

Length of Stay Prediction in Neurosurgery with Russian GPT-3 Language Model Compared to Human Expectations.

Patients, relatives, doctors, and healthcare providers anticipate the evidence-based length of stay (LOS) prediction in neurosurgery. This study aimed to assess the quality of LOS prediction with the GPT3 language model upon the narrative medical records in neurosurgery comparing to doctors' and patients' expectations. We found no significant difference (p = 0.109) between doctors', patients', and model's predictions with neurosurgeons tending to be more accurate in prognosis. The modern neural network language models demonstrate feasibility in LOS prediction.

The Classification of Short Scientific Texts Using Pretrained BERT Model.

Automated text classification is a natural language processing (NLP) technology that could significantly facilitate scientific literature selection. A specific topical dataset of 630 article abstracts was obtained from the PubMed database. We proposed 27 parametrized options of PubMedBERT model and 4 ensemble models to solve a binary classification task on that dataset. Three hundred tests with resamples were performed in each classification approach. The best PubMedBERT model demonstrated F1-score = 0.857 while the best ensemble model reached F1-score = 0.853. We concluded that the short scientific texts classification quality might be improved using the latest state-of-art approaches.

Semiautomated Approach for Muscle Weakness Detection in Clinical Texts.

The automated detection of adverse events in medical records might be a cost-effective solution for patient safety management or pharmacovigilance. Our group proposed an information extraction algorithm (IEA) for detecting adverse events in neurosurgery using documents written in a natural rich-in-morphology language. In this paper, we challenge to optimize and evaluate its performance for the detection of any extremity muscle weakness in clinical texts. Our algorithm shows the accuracy of 0.96 and ROC AUC = 0.96 and might be easily implemented in other medical domains.

The Classification of Scientific Literature for Its Topical Tracking on a Small Human-Prepared Dataset.

The number of scientific publications is constantly growing to make their processing extremely time-consuming. We hypothesized that a user-defined literature tracking may be augmented by machine learning on article summaries. A specific dataset of 671 article abstracts was obtained and nineteen binary classification options using machine learning (ML) techniques on various text representations were proposed in a pilot study. 300 tests with resamples were performed for each classification option. The best classification option demonstrated AUC = 0.78 proving the concept in general and indicating a potential for solution improvement.

Classification of Intracranial Hemorrhage Subtypes Using Deep Learning on CT Scans.

Intracranial hemorrhage is a pathological condition that requires fast diagnosis and decision making. Recently, a neural network model for classification of different intracranial hemorrhage types was proposed by a member of our research group Konstantin Kotik as part of the machine learning competition at Kaggle. Our current pilot study aimed to test this model on real-world CT scans from patients with intracranial hemorrhage treated at N.N. Burdenko Neurosurgery Center. The deep learning model for intracranial hemorrhage classification based on ResNexT architecture showed an accuracy of detection greater than 0.81 for every subtype of hemorrhage without any tuning. We expect further improvement in the model performance.

Detection of Muscle Weakness in Medical Texts Using Natural Language Processing.

Identifying adverse events in clinical documents is demanded in retrospective clinical research and prospective monitoring of treatment safety and cost-effectiveness. We proposed and evaluated a few methods of semi-automated muscle weakness detection in preoperative clinical notes for a larger project on predicting paresis by images. The combination of semi-expert and machine learning methods demonstrated maximized sensitivity = 0.860 and specificity = 0.919, and largest AUC = 0.943 with a 95% CI [0.874; 0.991], outperforming each method used individually. Our approaches are expected to be effective for autoshaping a well- verified training dataset for supervised machine learning.

Predicting Postoperative Hospital Stay in Neurosurgery with Recurrent Neural Networks Based on Operative Reports.

This study aimed to predict the duration of the postoperative in-hospital period in neurosurgery based on unstructured operative reports, natural language processing, and deep learning. The recurrent neuronal network (RNN-GRU) was tuned on the word-embedded reports of primary surgical cases retrieved for the period between 2000 and 2017. A new test dataset obtained for the primary operations performed in 2018-2019 was used to evaluate model performance. The mean absolute error of prediction in the final test was 3.00 days. Our study demonstrated the usability of textual EHRs data for the prediction of postoperative period length in neurosurgery using deep learning.

Prediction of Postoperative Hospital Stay with Deep Learning Based on 101 654 Operative Reports in Neurosurgery.

Electronic Health Records (EHRs) conceal a hidden knowledge that could be mined with data science tools. This is relevant for N.N. Burdenko Neurosurgery Center taking the advantage of a large EHRs archive collected for a period between 2000 and 2017. This study was aimed at testing the informativeness of neurosurgical operative reports for predicting the duration of postoperative stay in a hospital using deep learning techniques. The recurrent neuronal networks (GRU) were applied to the word-embedded texts in our experiments. The mean absolute error of prediction in 90% of cases was 2.8 days. These results demonstrate the potential utility of narrative medical texts as a substrate for decision support technologies in neurosurgery.