An open-source fine-tuned large language model for radiological impression generation: a multi-reader performance study.

BACKGROUND: The impression section integrates key findings of a radiology report but can be subjective and variable. We sought to fine-tune and evaluate an open-source Large Language Model (LLM) in automatically generating impressions from the remainder of a radiology report across different imaging modalities and hospitals. METHODS: In this institutional review board-approved retrospective study, we collated a dataset of CT, US, and MRI radiology reports from the University of California San Francisco Medical Center (UCSFMC) (n = 372,716) and the Zuckerberg San Francisco General (ZSFG) Hospital and Trauma Center (n = 60,049), both under a single institution. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score, an automatic natural language evaluation metric that measures word overlap, was used for automatic natural language evaluation. A reader study with five cardiothoracic radiologists was performed to more strictly evaluate the model's performance on a specific modality (CT chest exams) with a radiologist subspecialist baseline. We stratified the results of the reader performance study based on the diagnosis category and the original impression length to gauge case complexity. RESULTS: The LLM achieved ROUGE-L scores of 46.51, 44.2, and 50.96 on UCSFMC and upon external validation, ROUGE-L scores of 40.74, 37.89, and 24.61 on ZSFG across the CT, US, and MRI modalities respectively, implying a substantial degree of overlap between the model-generated impressions and impressions written by the subspecialist attending radiologists, but with a degree of degradation upon external validation. In our reader study, the model-generated impressions achieved overall mean scores of 3.56/4, 3.92/4, 3.37/4, 18.29 s,12.32 words, and 84 while the original impression written by a subspecialist radiologist achieved overall mean scores of 3.75/4, 3.87/4, 3.54/4, 12.2 s, 5.74 words, and 89 for clinical accuracy, grammatical accuracy, stylistic quality, edit time, edit distance, and ROUGE-L score respectively. The LLM achieved the highest clinical accuracy ratings for acute/emergent findings and on shorter impressions. CONCLUSIONS: An open-source fine-tuned LLM can generate impressions to a satisfactory level of clinical accuracy, grammatical accuracy, and stylistic quality. Our reader performance study demonstrates the potential of large language models in drafting radiology report impressions that can aid in streamlining radiologists' workflows.

Arterial Spin-Labeling MR Imaging in the Detection of Intracranial Arteriovenous Malformations in Patients with Hereditary Hemorrhagic Telangiectasia.

BACKGROUND AND PURPOSE: Hereditary hemorrhagic telangiectasia (HHT) is an autosomal dominant disease that causes vascular malformations in a variety of organs and tissues, including brain AVMs. Because brain AVMs have the potential to cause disabling or fatal intracranial hemorrhage, detection of these lesions before rupture is the goal of screening MR imaging/MRA examinations in patients with HHT. Prior studies have demonstrated superior sensitivity for HHT-related brain AVMs by using postcontrast MR imaging sequences as compared with MRA alone. We now present data regarding the incremental benefit of including arterial spin-labeling (ASL) perfusion sequences as part of MR imaging/MRA screening in patients with this condition. MATERIALS AND METHODS: We retrospectively analyzed 831 patients at the UCSF Hereditary Hemorrhagic Telangiectasia Center of Excellence. Of these, 42 patients had complete MR imaging/MRA, ASL perfusion scans, and criterion-standard DSA data. Two neuroradiologists reviewed imaging studies and a third provided adjudication when needed. RESULTS: Eight patients had no brain AVMs detected on DSA. The remaining 34 patients had 57 brain AVMs on DSA. Of the 57 identified AVMs, 51 (89.5%) were detected on ASL and 43 (75.4%) were detected on conventional MR imaging/MRA sequences (P = .049), with 8 lesions detected on ASL perfusion but not on conventional MR imaging. CONCLUSIONS: ASL provides increased sensitivity for brain AVMs in patients with HHT. Inclusion of ASL should be considered as part of comprehensive MR imaging/MRA screening protocols for institutions taking care of patients with HHT.

Application of a Domain-specific BERT for Detection of Speech Recognition Errors in Radiology Reports.

Purpose: To develop radiology domain-specific bidirectional encoder representations from transformers (BERT) models that can identify speech recognition (SR) errors and suggest corrections in radiology reports. Materials and Methods: A pretrained BERT model, Clinical BioBERT, was further pretrained on a corpus of 114 008 radiology reports between April 2016 and August 2019 that were retrospectively collected from two hospitals. Next, the model was fine-tuned on a training dataset of generated insertion, deletion, and substitution errors, creating Radiology BERT. This model was retrospectively evaluated on an independent dataset of radiology reports with generated errors (n = 18 885) and on unaltered report sentences (n = 2000) and prospectively evaluated on true clinical SR errors (n = 92). Correction Radiology BERT was separately trained to suggest corrections for detected deletion and substitution errors. Area under the receiver operating characteristic curve (AUC) and bootstrapped 95% CIs were calculated for each evaluation dataset. Results: Radiology-specific BERT had AUC values of >.99 (95% CI: >0.99, >0.99), 0.94 (95% CI: 0.93, 0.94), 0.98 (95% CI: 0.98, 0.98), and 0.97 (95% CI: 0.97, 0.97) for detecting insertion, deletion, substitution, and all errors, respectively, on the independently generated test set. Testing on unaltered report impressions revealed a sensitivity of 82% (28 of 34; 95% CI: 70%, 93%) and specificity of 88% (1521 of 1728; 95% CI: 87%, 90%). Testing on prospective SR errors showed an accuracy of 75% (69 of 92; 95% CI: 65%, 83%). Finally, the correct word was the top suggestion for 45.6% (475 of 1041; 95% CI: 42.5%, 49.3%) of errors. Conclusion: Radiology-specific BERT models fine-tuned on generated errors were able to identify SR errors in radiology reports and suggest corrections.Keywords: Computer Applications, Technology Assessment Supplemental material is available for this article. © RSNA, 2022See also the commentary by Abajian and Cheung in this issue.

Characteristics and Growth Rate of Lung Metastases in Patients With Primary Gastrointestinal Malignancies and Lung-dominant Metastatic Disease: A Retrospective Cohort Analysis.

INTRODUCTION: There are no formal guidelines for the management of patients with primary gastrointestinal (GI) cancers who have lung-exclusive or lung-predominant metastases. We performed a retrospective analysis to evaluate host and tumor characteristics of this patient population, model patterns and rates of growth, and describe treatment approaches. MATERIALS AND METHODS: Eligible patients had a GI cancer with either synchronous or metachronous lung metastases but no other visceral or peritoneal sites of involvement. In addition to collecting detailed patient-specific and tumor-specific information, all imaging studies (computed tomography±positron emission tomography scans) were reviewed by an independent radiologist. Up to 5 lung metastases were tracked through each patient's clinical course. Growth rate was estimated using a linear mixed model analysis. RESULTS: Forty patients met eligibility criteria (18 pancreatic, 15 colorectal, 6 hepatobiliary, 1 gastroesophageal; synchronous vs. metachronous, 13 and 27, respectively). Median time from original cancer diagnosis to onset of metachronous lung lesions was 16 months. Interval from first appearance of lung metastases to treatment initiation was 6.2 months. Average growth rate of the largest lesion was 0.21 mm/mo (95% confidence interval, 0.12-0.30), with substantial intrapatient and interpatient variability. Sixty percent of patients underwent locoregional interventions in addition to or in lieu of systemic therapy for their lung metastases. Median survival of the entire study cohort from first appearance of lung metastases was 54 months. CONCLUSIONS: Lung metastases from primary GI cancers have a variable but overall indolent natural history and are generally associated with prolonged survival outcomes. Further efforts to define patterns of growth of lung metastases, informed by size, number, and clinical/molecular features, are needed to guide appropriate timing and selection of therapy as well as surveillance strategies.

Are Women Disproportionately Represented in Education Compared to Other Roles in Academic Radiology?

RATIONALE AND OBJECTIVES: Women in academic medicine, and radiology specifically, are underrepresented in departmental leadership roles and achieve fewer professional metrics of success. We have observed, however, that women are more broadly represented in medical education leadership. The purpose of this study was to determine if women in academic radiology are overrepresented in educational scholarship and educational leadership compared to general research scholarship and leadership positions, and to determine if there is any difference in the distribution of women in these roles compared to internal medicine. MATERIALS AND METHODS: We performed a PubMed search of education articles in radiology and internal medicine over the last 5 years, and the gender of the authors was determined. Data on gender for authorship of general research topics, departmental leaders, and society leadership was obtained via literature and internet searches and the data was aggregated. Representation ratios (RR of 1.0 = parity) were obtained via risk ratio calculation to compare education versus general scholarship, and the distribution of leadership roles within and between these fields. RESULTS: Women make up 28.5% of academic radiologists and 40.1% of academic internists. A higher proportion of education articles were first authored by women than would be expected in both fields with an RR of 1.46 (p < 0.001) in radiology and 1.23 (p < 0.001) in internal medicine. This overrepresentation was significant compared to general research scholarship in both fields (p < 0.001). In both fields, women were overrepresented in the position of Medical Student Director (RR of 1.47 and 1.22, respectively). For Program Directors, women were overrepresented in radiology (RR of 1.12) and underrepresented in internal medicine (RR of 0.69-0.75). Women in radiology were overrepresented in education society leadership (RR = 1.63) compared to general society leadership (0.98, p = 0.001). CONCLUSION: We found that women in radiology are overrepresented in authorship of education articles, and in departmental and society education leadership roles. This trend was also seen in internal medicine, suggesting that women are more broadly represented in medical education.

Association of Receipt of Positron Emission Tomography-Computed Tomography With Non-Small Cell Lung Cancer Mortality in the Veterans Affairs Health Care System.

Importance: Positron emission tomography-computed tomography (PET-CT) has been increasingly used in the management of lung cancer, but its association with survival has not been convincingly documented. Objective: To examine the association of the use of PET-CT with non-small cell lung cancer (NSCLC) mortality in the US Department of Veterans Affairs (VA) health care system from 2000 to 2013. Design, Setting, and Participants: This cohort study included 64 103 veterans receiving care in the VA health care system who were diagnosed with incident NSCLC between September 2000 and December 2013. Data analysis took place in October 2018. Exposure: Use of PET-CT before and/or after diagnosis. Main Outcomes and Measures: All-cause and NSCLC-specific 5-year mortality; secondary outcome was receipt of stage-appropriate treatment. Results: A total of 64 103 veterans with the diagnosis of NSCLC were evaluated; 62 838 (98.0%) were men, and 50 584 (78.9%) were white individuals. Among these, 51 844 (80.9%) had a PET-CT performed: 25 735 (40.1%) in the 12 months before diagnosis and 41 242 (64.3%) in the 5 years after diagnosis. Increased PET-CT use (597 of 978 veterans [59.2%] in 2000 vs 3649 of 3915 [93.2%] in 2013) and decreased NSCLC-specific 5-year mortality (879 of 978 veterans [89.9%] in 2000 vs 3226 of 3915 veterans [82.4%] in 2013) were found over time. Increased use of stage-appropriate therapy was also seen over time, from 346 of 978 veterans (35.4%) in 2000 to 2062 of 3915 (52.7%) in 2013 (P < .001). Increased PET-CT use was associated with higher-complexity level VA facilities (26 127 veterans [82.3%] at level 1a vs 1289 [75.2%] at level 3 facilities; P < .001) and facilities with on-site PET-CT compared with facilities without on-site PET-CT (33 081 [82.2%] vs 17 443 [80.3%]; P < .001). Use of PET-CT before diagnosis was associated with increased likelihood of stage-appropriate treatment for all stages of NSCLC (eg, veterans with stage 1 disease: 4837 of 7870 veterans [61.5%] who received PET-CT underwent surgical resection vs 4042 of 7938 veterans [50.9%] who did not receive PET-CT; P < .001) and decreased mortality in a risk-adjusted model among all participants and among veterans undergoing stage-appropriate treatment (all-cause mortality: hazard ratio [HR], 0.78; 95% CI, 0.77-0.79; NSCLC-specific mortality: HR, 0.78; 95% CI, 0.76-0.80). Facilities with on-site PET-CT and higher-complexity level facilities were associated with a mortality benefit, with 16% decreased mortality at level 1a vs level 3 facilities (HR, 0.84; 95% CI, 0.80-0.89) and a 3% decrease in all-cause mortality in facilities with on-site PET-CT (HR, 0.97; 95% CI, 0.96-0.99). Conclusions: In this study, the use of PET-CT among veterans with NSCLC significantly increased from 2000 to 2013, coinciding with decreased 5-year mortality and an increase in stage-appropriate treatment. Variation in use of PET-CT was found, with the highest use at higher-complexity level facilities and those with PET-CT on-site. These facilities were associated with reduced all-cause and NSCLC-specific mortality.

Metastatic cervical paravertebral solitary fibrous tumor detected by fluorodeoxyglucose positron emission tomography-computed tomography.

Solitary fibrous tumor/hemangiopericytomas (SFT/HPC) are soft tissue tumors that can arising from the abomen, pleura, head and neck, or extremities. We report an unusual case of recurrent hemangiopericytoma in a 67-year-old female presenting with a painless and palpable mass within her right posterior neck. Eight years after initial resection of the mass, a follow-up MRI showed multiple enlarging calvarial lesions. A whole body FDG-PET/CT revealed not only hypermetabolic calvarial lesions but also numerous hypermetabolic axillary node and osseous metastases. Though the majority of these soft tissue tumors exhibit benign behavior and carry a favorable prognosis, patients with these slow growing tumors are at risk for local recurrence and distant metastases which demonstrate substantial FDG avidity. Additional studies are needed to clarify the role of whole body FDG-PET/CT in the surveillance of SFT/HPC to detect recurrent or metastatic lesions.

Interpretive versus noninterpretive content in top-selling radiology textbooks: what are we teaching medical students?

RATIONALE AND OBJECTIVES: There are little data as to whether appropriate, cost effective, and safe ordering of imaging examinations are adequately taught in US medical school curricula. We sought to determine the proportion of noninterpretive content (such as appropriate ordering) versus interpretive content (such as reading a chest x-ray) in the top-selling medical student radiology textbooks. MATERIALS AND METHODS: We performed an online search to identify a ranked list of the six top-selling general radiology textbooks for medical students. Each textbook was reviewed including content in the text, tables, images, figures, appendices, practice questions, question explanations, and glossaries. Individual pages of text and individual images were semiquantitatively scored on a six-level scale as to the percentage of material that was interpretive versus noninterpretive. The predominant imaging modality addressed in each was also recorded. Descriptive statistical analysis was performed. RESULTS: All six books had more interpretive content. On average, 1.4 pages of text focused on interpretation for every one page focused on noninterpretive content. Seventeen images/figures were dedicated to interpretive skills for every one focused on noninterpretive skills. In all books, the largest proportion of text and image content was dedicated to plain films (51.2%), with computed tomography (CT) a distant second (16%). The content on radiographs (3.1:1) and CT (1.6:1) was more interpretive than not. CONCLUSIONS: The current six top-selling medical student radiology textbooks contain a preponderance of material teaching image interpretation compared to material teaching noninterpretive skills, such as appropriate imaging examination selection, rational utilization, and patient safety.

Contrast staining on CT after DSA in ischemic stroke patients progresses to infarction and rarely hemorrhages.

Contrast staining of brain parenchyma identified on non-contrast CT performed after DSA in patients with acute ischemic stroke (AIS) is an incompletely understood imaging finding. We hypothesize contrast staining to be an indicator of brain injury and suspect the fate of involved parenchyma to be cerebral infarction. Seventeen years of AIS data were retrospectively analyzed for contrast staining. Charts were reviewed and outcomes of the stained parenchyma were identified on subsequent CT and MRI. Thirty-six of 67 patients meeting inclusion criteria (53.7%) had contrast staining on CT obtained within 72 hours after DSA. Brain parenchyma with contrast staining in patients with AIS most often evolved into cerebral infarction (81%). Hemorrhagic transformation was less likely in cases with staining compared with hemorrhagic transformation in the cohort that did not have contrast staining of the parenchyma on post DSA CT (6% versus 25%, respectively, OR 0.17, 95% CI 0.017 - 0.98, p = 0.02). Brain parenchyma with contrast staining on CT after DSA in AIS patients was likely to infarct and unlikely to hemorrhage.