Deep Learning-Based Natural Language Processing in Radiology: The Impact of Report Complexity, Disease Prevalence, Dataset Size, and Algorithm Type on Model Performance.

In radiology, natural language processing (NLP) allows the extraction of valuable information from radiology reports. It can be used for various downstream tasks such as quality improvement, epidemiological research, and monitoring guideline adherence. Class imbalance, variation in dataset size, variation in report complexity, and algorithm type all influence NLP performance but have not yet been systematically and interrelatedly evaluated. In this study, we investigate these factors on the performance of four types [a fully connected neural network (Dense), a long short-term memory recurrent neural network (LSTM), a convolutional neural network (CNN), and a Bidirectional Encoder Representations from Transformers (BERT)] of deep learning-based NLP. Two datasets consisting of radiologist-annotated reports of both trauma radiographs (n = 2469) and chest radiographs and computer tomography (CT) studies (n = 2255) were split into training sets (80%) and testing sets (20%). The training data was used as a source to train all four model types in 84 experiments (Fracture-data) and 45 experiments (Chest-data) with variation in size and prevalence. The performance was evaluated on sensitivity, specificity, positive predictive value, negative predictive value, area under the curve, and F score. After the NLP of radiology reports, all four model-architectures demonstrated high performance with metrics up to > 0.90. CNN, LSTM, and Dense were outperformed by the BERT algorithm because of its stable results despite variation in training size and prevalence. Awareness of variation in prevalence is warranted because it impacts sensitivity and specificity in opposite directions.

Improvement of radiology reporting in a clinical cancer network: impact of an optimised multidisciplinary workflow.

PURPOSE: To assess the effectiveness of implementing a quality improvement project in a clinical cancer network directed at the response assessment of oncology patients according to RECIST-criteria. METHODS: Requests and reports of computed tomography (CT) studies from before (n = 103) and after (n = 112) implementation of interventions were compared. The interventions consisted of: a multidisciplinary working agreement with a clearly described workflow; subspecialisation of radiologists; adaptation of the Picture Archiving and Communication System (PACS); structured reporting. RESULTS: The essential information included in the requests and the reports improved significantly after implementation of the interventions. In the requests, mentioning start date increased from 2% to 49%; date of baseline CT from 7% to 64%; nadir date from 1% to 41%. In the reports, structured layout increased from 14% to 86%; mentioning target lesions from 18% to 80% and non-target lesions from 11% to 80%; measurements stored in PACS increased from 76% to 97%; labelled key images from 38% to 95%; all p values < 0.001. CONCLUSION: The combination of implementation of an optimised workflow, subspecialisation and structured reporting led to significantly better quality radiology reporting for oncology patients receiving chemotherapy. The applied multifactorial approach can be used within other radiology subspeciality areas as well. KEY POINTS: • Undeveloped subspecialisation makes adherence to RECIST guidelines difficult in general hospitals. • A clinical cancer network provides opportunities to improve healthcare. • Optimised workflow, subspecialisation and structured reporting substantially improve request and report quality. • Good interdisciplinary communication between oncologists, radiologists and others contributes to quality improvement.

Perception of radiology reporting efficacy by neurologists in general and university hospitals.

AIM: To investigate how neurologists perceive the value of the radiology report and to analyse the relation with the neurologists own expertise in radiology and the level of subspecialisation of radiologists. MATERIALS AND METHODS: A web-based survey was distributed to neurologists. The level of subspecialisation was assessed by the percentage of fellowship-trained radiologists and the percentage of radiologists that were members of the Dutch Society of Neuroradiology. RESULTS: Most neurologists interpret all computed tomography (CT) and magnetic resonance imaging (MRI) studies themselves, and their self-confidence in making correct interpretations is high. Residents gave higher scores than neurologists for "Radiologist report answers the question" (p=0.039) and for "Radiologist reports give helpful advice" (p=0.001). Neurologists from university hospitals stated more frequently that the report answered their questions than neurologists from general hospitals (p=0.008). The general appreciation for radiology reports was higher for neurologists from university hospitals than from general hospitals (8.2 versus 7.2; p=0.003). Radiologists at university hospitals have a higher level of subspecialisation than those at general hospitals. CONCLUSION: Subspecialisation of radiologists leads to higher quality of radiology reporting as perceived by neurologists. Because of their expertise in radiology, neurologists are valuable sources of feedback for radiologists. Paying attention to the clinical questions and giving advice tailored to the needs of the referring physicians are opportunities to improve radiology reporting.

Implementation and Validation of PACS Integrated Peer Review for Discrepancy Recording of Radiology Reporting.

The purpose of this work is to demonstrate the possibility of implementation of a PACS-integrated peer review system based on RADPEER™ classification providing a step-wise implementation plan utilizing features already present in the standard PACS implementation and without the requirement of additional software development. Furthermore, we show the usage and effects of the system during the first 30 months of usage. To allow fast and easy implementation into the daily workflow the key-word feature of the PACS was used. This feature allows to add a key-word to an imaging examination for easy searching in the PACS database (e.g. by entering keywords for different kinds of pathology). For peer review we implemented a keyword structure including a code for each of the existing RADPEER™ scoring language terms and a keyword with the phrase "second reading" followed by the name of the individual radiologist. The use of the short-keys to enter the codes in relation to the peer review was a simple to use solution. During the study 599 reports were peer reviewed. The active participation in this study of the radiologists varies and ranges from 3 to 327 reviews per radiologist. The number of peer review is highest in CT and CR. There are no significant technical obstacles to implement a PACS-integrated RADPEER™ -system based on key-words allowing easy integration of peer review into the daily routine without the requirement of additional software. Peer review implemented in a non-random setting based on relevant priors could already help in increasing the quality of radiological reporting and serve as continuing education among peers. Decisiveness, tact and trust are needed to promote use of the system and collaborative discussion of the results by radiologist.

Machine learning based natural language processing of radiology reports in orthopaedic trauma.

OBJECTIVES: To compare different Machine Learning (ML) Natural Language Processing (NLP) methods to classify radiology reports in orthopaedic trauma for the presence of injuries. Assessing NLP performance is a prerequisite for downstream tasks and therefore of importance from a clinical perspective (avoiding missed injuries, quality check, insight in diagnostic yield) as well as from a research perspective (identification of patient cohorts, annotation of radiographs). METHODS: Datasets of Dutch radiology reports of injured extremities (n = 2469, 33% fractures) and chest radiographs (n = 799, 20% pneumothorax) were collected in two different hospitals and labeled by radiologists and trauma surgeons for the presence or absence of injuries. NLP classification was applied and optimized by testing different preprocessing steps and different classifiers (Rule-based, ML, and Bidirectional Encoder Representations from Transformers (BERT)). Performance was assessed by F1-score, AUC, sensitivity, specificity and accuracy. RESULTS: The deep learning based BERT model outperforms all other classification methods which were assessed. The model achieved an F1-score of (95 ± 2)% and accuracy of (96 ± 1)% on a dataset of simple reports (n= 2469), and an F1 of (83 ± 7)% with accuracy (93 ± 2)% on a dataset of complex reports (n= 799). CONCLUSION: BERT NLP outperforms traditional ML and rule-base classifiers when applied to Dutch radiology reports in orthopaedic trauma.

Judgements of ordinality and summation of number symbols by squirrel monkeys (Saimiri sciureus).

In Experiment 1, 2 squirrel monkeys (Saimiri sciureus) were given choices between all possible pairs of the arabic numbers 0, 1, 3, 5, 7, and 9, with choice of any number yielding that number of pieces of peanut as a reward. Both monkeys learned to choose the larger number in all pairings and learned to choose the largest number within a set of 4 numbers. In Experiments 2-4, the monkeys were tested on problems in which they chose between pairs of stimuli containing 2 numbers versus 2 numbers, 1 number versus 2 numbers, and 3 numbers versus 3 numbers. Both monkeys showed a significant tendency to choose the stimulus that contained the largest sum. Various tests indicated that this effect could not be explained by choice of the stimulus with the largest single number, by avoidance of the stimulus with the smallest single number, or by experimenter cuing.

Timing and classifying brief acoustic stimuli by songbirds and humans.

The durations of animals' brief vocalizations provide conspecifics with important recognition cues. In the present experiments, zebra finches and humans (trained musicians) were rewarded for responding after S+ (standard) auditory signals from 56 to 663 ms and not for responding after shorter or longer S- (comparison) durations from 10 to 3684 ms. With either a single standard (Experiment 1) or multiple standards (Experiment 2), both zebra finches and humans timed brief signals to about the same level of accuracy. The results were in qualitative agreement with predictions from scalar timing theory and its connectionist implementation in both experiments. The connectionist model provides a good quantitative account of temporal gradients with a single standard (Experiment 1) but not with multiple standards (Experiment 2).

Summation of symbols by pigeons (Columba livia): the importance of number and mass of reward items.

In each of 3 experiments, different sets of 4 pigeons (Columba livia) were trained to discriminate between 2 visual symbols that covered wells containing food items that varied in number, mass, or both. In Experiment 1, the symbols were associated with 0, 1, 3, 5, 7, or 9 pieces of grain reward. The pigeons learned to choose the symbol corresponding to the larger reward, and on summation tests, they chose the pair of symbols that summed to the larger total reward. When number of food pellets was varied but mass of reward was held constant in Experiment 2, preference for the larger number symbols failed to appear. When number was held constant and mass was varied in Experiment 3, the pigeons showed a clear preference for the larger mass symbols on single-symbol and summation tests. These findings show that pigeons summate the value of symbols and are more likely to represent symbols by mass of food reward than by number of food items.

Osteoblastic differentiation.

The fully differentiated osteoblast may be easily recognized in bone tissue. Its cuboidal shape, its position directly opposed to the bone surface and its capacity to produce calcified bone matrix are characteristic. Three other differentiation stages are also reasonably well defined--the preosteoblast, the osteocyte and the lining cell. These differentiation stages are preceded by an unknown number of precursor, progenitor and stem cell stages. Little is known about the regulation of the transitions between the various osteogenic phenotypes and their reversibility or irreversibility. One of the reasons for this is the lack of adequate tools with which to recognize the various differentiation stages. We have developed a number of monoclonal antibodies (in bone) specifically directed against osteocytes, osteoblasts and as yet unidentified cells in the periosteum. The anti-osteocyte monoclonals were used to recognize osteocytes in bone cell cultures and we obtained purified osteocyte populations for metabolic studies. Osteocytes were shown to have binding sites for parathyroid hormone. The antibodies directed against osteoblasts showed that at present our culture conditions are inadequate to allow osteoblast differentiation in vitro.

Induction of ornithine decarboxylase activity in isolated chicken osteoblasts by parathyroid hormone: the role of cAMP and calcium.

We investigated the role of cAMP and Ca2+ as mediators in parathyroid hormone (PTH)-induced ornithine decarboxylase (ODC) activity in primary cultures of chicken osteoblasts. We present evidence that the induction of ODC activity by PTH is most likely a receptor-mediated process and that cAMP is a mediator. However, using three different approaches we have strong indications that cAMP is not the exclusive mediator of PTH-induced ODC activity. First, when the dose-response curve of PTH-induced ODC activity is compared with that of PTH-stimulated cAMP production, the ED50 for cAMP production is about five times as high as that for the induction of ODC activity. Second, 1 mM 9-(tetrahydro-2-furanyl) adenine (SQ 22.536) almost completely inhibited PTH-stimulated cAMP production whereas there was only a small inhibitory effect on PTH-induced ODC activity. Third, some PTH fragments unable to stimulate cAMP production were still able to induce ODC activity. We therefore propose that apart from cAMP, an additional messenger, most likely Ca2+, must be present. Evidence for this concept are the observations that substances affecting extracellular and intracellular Ca2+ levels (EGTA, A23187, CoCl2, verapamil) or antagonizing calmodulin (Trifluoroperazin, Compound 48/80) also strongly affect PTH-induced ODC activity. These effects could not be explained by a positive interaction of Ca2+ with the hormone-stimulated cAMP system as 2 mM EGTA strongly enhanced PTH-stimulated cAMP production but at the same time completely inhibited PTH-induced ODC activity. A similar dissociation between hormone-induced cAMP production and induction of ODC activity was found with the Ca2+ -ionophore A23187 (10(-7) M) which significantly inhibited PTH-stimulated cAMP production but strongly enhanced PTH-induced ODC activity. Our results suggest that intracellular Ca2+, and possibly calmodulin, in addition to cAMP, are involved in PTH-induced ODC activity in chicken osteoblasts. Most probably Ca2+ is the initial messenger and cAMP acts in a coordinate pattern as a synarchic messenger making the induction of ODC activity by PTH more sensitive to Ca2+. Furthermore, the present findings are in agreement with our concept of the existence of two receptors or two receptor-sites for PTH on osteoblasts. One receptor is coupled to the production of cAMP and is presumably activated when the first two aminoacids of the NH2-terminus of the hormone are present and the other, suggested to be responsible for the increase in intracellular Ca2+, is thought to be activated by a region of the hormone sequence between amino acid 3 and 34.(ABSTRACT TRUNCATED AT 250 WORDS)