Do Neural Information Extraction Algorithms Generalize Across Institutions?

PURPOSE: Natural language processing (NLP) techniques have been adopted to reduce the curation costs of electronic health records. However, studies have questioned whether such techniques can be applied to data from previously unseen institutions. We investigated the performance of a common neural NLP algorithm on data from both known and heldout (ie, institutions whose data were withheld from the training set and only used for testing) hospitals. We also explored how diversity in the training data affects the system's generalization ability. METHODS: We collected 24,881 breast pathology reports from seven hospitals and manually annotated them with nine key attributes that describe types of atypia and cancer. We trained a convolutional neural network (CNN) on annotations from either only one (CNN1), only two (CNN2), or only four (CNN4) hospitals. The trained systems were tested on data from five organizations, including both known and heldout ones. For every setting, we provide the accuracy scores as well as the learning curves that show how much data are necessary to achieve good performance and generalizability. RESULTS: The system achieved a cross-institutional accuracy of 93.87% when trained on reports from only one hospital (CNN1). Performance improved to 95.7% and 96%, respectively, when the system was trained on reports from two (CNN2) and four (CNN4) hospitals. The introduction of diversity during training did not lead to improvements on the known institutions, but it boosted performance on the heldout institutions. When tested on reports from heldout hospitals, CNN4 outperformed CNN1 and CNN2 by 2.13% and 0.3%, respectively. CONCLUSION: Real-world scenarios require that neural NLP approaches scale to data from previously unseen institutions. We show that a common neural NLP algorithm for information extraction can achieve this goal, especially when diverse data are used during training.

Current state of practice regarding digital radiography exposure indicators and deviation indices: Report of AAPM Imaging Physics Committee Task Group 232.

Beginning with the advent of digital radiography systems in 1981, manufacturers of these systems provided indicators of detector exposure. These indicators were manufacturer-specific, and users in facilities with equipment from multiple manufacturers found it a challenge to monitor and manage variations in indicated exposure in routine clinical use. In 2008, a common definition of exposure index (EI) was realized in International Electrotechnical Commission (IEC) International Standard 62494-1 Ed. 1, which also introduced and defined the deviation index (DI), a number quantifying the difference between the detector EI for a given radiograph and the target exposure index (EIT ). An exposure index that differed by a constant from that established by the IEC and the concept of the deviation index also appear in American Association of Physicists in Medicine (AAPM) Report No. 116 published in 2009. The AAPM Report No. 116 went beyond the IEC standard in supplying a table (Table II in the report of TG-116) titled "Exposure Indicator DI Control Limits for Clinical Images," which listed suggested DI ranges and actions to be considered for each range. As the IEC EI was implemented and clinical DI data were gathered, concerns were voiced that the DI control limits published in the report of TG-116 were too strict and did not accurately reflect clinical practice. The charge of task group 232 (TG-232) and the objective of this final report was to investigate the current state of the practice for CR/DR Exposure and Deviation Indices based on AAPM TG 116 and IEC-62494, for the purpose of establishing achievable goals (reference levels) and action levels in digital radiography. Data corresponding to EI and DI were collected from a range of practice settings for a number of body parts and views (adults and pediatric radiographs) and analyzed in aggregate and separately. A subset of radiographs was also evaluated by radiologists based on criteria adapted from the European Guidelines on Quality Criteria for Diagnostic Radiographic Images from the European Commission. Analysis revealed that typical DI distribution was characterized by a standard deviation (SD) of 1.3-3.6 with mean DI values substantially different from 0.0, and less than 50% of DI values fell within the significant action limits proposed by AAPM TG-116 (-1.0 ≤ DI ≤ 1.0). Recommendations stemming from this analysis include targeting a mean DI value of 0.0 and action limits at ±1 and ±2 SD of the DI based on actual DI data of an individual site. EIT values, DI values, and associated action limits should be reviewed on an ongoing basis and optimization of DI values should be a process of continuous quality improvement with a goal of reducing practice variation.

Ongoing quality control in digital radiography: Report of AAPM Imaging Physics Committee Task Group 151.

Quality control (QC) in medical imaging is an ongoing process and not just a series of infrequent evaluations of medical imaging equipment. The QC process involves designing and implementing a QC program, collecting and analyzing data, investigating results that are outside the acceptance levels for the QC program, and taking corrective action to bring these results back to an acceptable level. The QC process involves key personnel in the imaging department, including the radiologist, radiologic technologist, and the qualified medical physicist (QMP). The QMP performs detailed equipment evaluations and helps with oversight of the QC program, the radiologic technologist is responsible for the day-to-day operation of the QC program. The continued need for ongoing QC in digital radiography has been highlighted in the scientific literature. The charge of this task group was to recommend consistency tests designed to be performed by a medical physicist or a radiologic technologist under the direction of a medical physicist to identify problems with an imaging system that need further evaluation by a medical physicist, including a fault tree to define actions that need to be taken when certain fault conditions are identified. The focus of this final report is the ongoing QC process, including rejected image analysis, exposure analysis, and artifact identification. These QC tasks are vital for the optimal operation of a department performing digital radiography.

Essentials and guidelines for clinical medical physics residency training programs: executive summary of AAPM Report Number 249.

There is a clear need for established standards for medical physics residency training. The complexity of techniques in imaging, nuclear medicine, and radiation oncology continues to increase with each passing year. It is therefore imperative that training requirements and competencies are routinely reviewed and updated to reflect the changing environment in hospitals and clinics across the country. In 2010, the AAPM Work Group on Periodic Review of Medical Physics Residency Training was formed and charged with updating AAPM Report Number 90. This work group includes AAPM members with extensive experience in clinical, professional, and educational aspects of medical physics. The resulting report, AAPM Report Number 249, concentrates on the clinical and professional knowledge needed to function independently as a practicing medical physicist in the areas of radiation oncology, imaging, and nuclear medicine, and constitutes a revision to AAPM Report Number 90. This manuscript presents an executive summary of AAPM Report Number 249.

An exposure indicator for digital radiography: AAPM Task Group 116 (executive summary).

Digital radiographic imaging systems, such as those using photostimulable storage phosphor, amorphous selenium, amorphous silicon, CCD, and MOSFET technology, can produce adequate image quality over a much broader range of exposure levels than that of screen/film imaging systems. In screen/film imaging, the final image brightness and contrast are indicative of over- and underexposure. In digital imaging, brightness and contrast are often determined entirely by digital postprocessing of the acquired image data. Overexposure and underexposures are not readily recognizable. As a result, patient dose has a tendency to gradually increase over time after a department converts from screen/film-based imaging to digital radiographic imaging. The purpose of this report is to recommend a standard indicator which reflects the radiation exposure that is incident on a detector after every exposure event and that reflects the noise levels present in the image data. The intent is to facilitate the production of consistent, high quality digital radiographic images at acceptable patient doses. This should be based not on image optical density or brightness but on feedback regarding the detector exposure provided and actively monitored by the imaging system. A standard beam calibration condition is recommended that is based on RQA5 but uses filtration materials that are commonly available and simple to use. Recommendations on clinical implementation of the indices to control image quality and patient dose are derived from historical tolerance limits and presented as guidelines.

Reliability of tumor volume estimation from MR images in patients with malignant glioma. Results from the American College of Radiology Imaging Network (ACRIN) 6662 Trial.

Reliable assessment of tumor growth in malignant glioma poses a common problem both clinically and when studying novel therapeutic agents. We aimed to evaluate two software-systems in their ability to estimate volume change of tumor and/or edema on magnetic resonance (MR) images of malignant gliomas. Twenty patients with malignant glioma were included from different sites. Serial post-operative MR images were assessed with two software systems representative of the two fundamental segmentation methods, single-image fuzzy analysis (3DVIEWNIX-TV) and multi-spectral-image analysis (Eigentool), and with a manual method by 16 independent readers (eight MR-certified technologists, four neuroradiology fellows, four neuroradiologists). Enhancing tumor volume and tumor volume plus edema were assessed independently by each reader. Intraclass correlation coefficients (ICCs), variance components, and prediction intervals were estimated. There were no significant differences in the average tumor volume change over time between the software systems (p > 0.05). Both software systems were much more reliable and yielded smaller prediction intervals than manual measurements. No significant differences were observed between the volume changes determined by fellows/neuroradiologists or technologists.Semi-automated software systems are reliable tools to serve as outcome parameters in clinical studies and the basis for therapeutic decision-making for malignant gliomas, whereas manual measurements are less reliable and should not be the basis for clinical or research outcome studies.

Reporter gene imaging using radiographic contrast from nonradioactive iodide sequestered by the sodium-iodide symporter.

The hypothesis that the human sodium-iodide symporter, NIS, can be used to detect NIS expression using standard radiological techniques was tested using adenoviral transduced NIS expression in human tumor xenografts grown in mice and in a naive dog prostate. Nonradioactive iodide was administered systemically to animals that 1-3 days previously had received a local injection of a replication-competent adenovirus expressing NIS under the control of the CMV promoter. The distribution of radiopacity was assessed in mouse tumors using micro-CT and a clinical X-ray machine and in the prostate of an anesthetized dog using a clinical spiral CT. Iodide sequestration and NIS expression were measured using X-ray spectrochemical analysis and fluorescence microscopy, respectively. Radiographic contrast due to NIS gene expression that was observed indicates the technique has potential for use in preclinical rodent tumor studies but probably lacks sensitivity for human use.

Magnetically-labeled sensitized splenocytes to identify glioma by MRI: a preliminary study.

This study investigated the feasibility of imaging the migration and incorporation of magnetically-labeled sensitized splenocytes in an experimental 9L glioma brain tumor model. Splenocytes collected from tumor-bearing (sensitized splenocytes) or control (nonsensitized splenocytes) host rats were analyzed to determine the population of different cells, labeled with ferumoxides-protamine sulfate (FePro) and injected intravenously to recipient rats (N=4, for each group) bearing intracranial 9L tumors. Day 3 postinjection of splenocytes multiecho T2*-weighted and three-dimensional (3D) gradient echo MRI were obtained using a 7 Tesla MR system. R2* (1/T2*) maps were created from the T2*-weighted images. Signal intensities (SIs) and R2* values in the tumors and contralateral brain were determined by hand drawn regions of interest (ROIs). Brain sections were stained for the evidence of administered cells. Both 3D and T2*-weighted MRI showed low signal intensity areas in and around the tumors in rats that received labeled sensitized splenocytes. Prussian blue (PB), CD45- and CD8-positive cells were present in areas at the corresponding sites of low signal intensities seen on MRI. Rats that received labeled nonsensitized splenocytes did not show low signal intensity areas or PB positive cells in or around the implanted tumors. In conclusion, the immunogenic reaction can be exploited to delineate recurrent glioma using MRI following systemically delivered magnetically labeled sensitized splenocytes or T-cells.

Model-independent method for fMRI analysis.

This paper presents a fast method for delineation of activated areas of the brain from functional magnetic resonance imaging (fMRI) time series data. The steps of the work accomplished are as follows. 1) It is shown that the detection performance evaluated by the area under the receiver operating characteristic curve is directly related to the signal-to-noise ratio (SNR) of the composite image generated in the detection process. 2) Detection and segmentation of activated areas are formulated in a vector space framework. In this formulation, a linear transformation (image combination method) is shown to be desirable to maximize the SNR of the activated areas subject to the constraint of removing inactive areas. 3) An analytical solution for the problem is found. 4) Image pixel vectors and expected time series pattern (signature) for inactive pixels are used to calculate weighting vector and identify activated regions. 5) Signatures of the activated regions are used to segment different activities. 6) Segmented images by the proposed method are compared with those generated by the conventional methods (correlation, t-statistic, and z statistic). Detection performance and SNRs of the images are compared. The proposed approach outperforms the conventional methods of fMRI analysis. In addition, it is model-independent and does not require a priori knowledge of the fMRI response to the paradigm. Since the method is linear and most of the work is done analytically, numerical implementation and execution of the method are much faster than the conventional methods.