Cardiac valve calcifications on low-dose unenhanced ungated chest computed tomography: inter-observer and inter-examination reliability, agreement and variability.

OBJECTIVES: To determine inter-observer and inter-examination variability for aortic valve calcification (AVC) and mitral valve and annulus calcification (MC) in low-dose unenhanced ungated lung cancer screening chest computed tomography (CT). METHODS: We included 578 lung cancer screening trial participants who were examined by CT twice within 3 months to follow indeterminate pulmonary nodules. On these CTs, AVC and MC were measured in cubic millimetres. One hundred CTs were examined by five observers to determine the inter-observer variability. Reliability was assessed by kappa statistics (κ) and intra-class correlation coefficients (ICCs). Variability was expressed as the mean difference ± standard deviation (SD). RESULTS: Inter-examination reliability was excellent for AVC (κ = 0.94, ICC = 0.96) and MC (κ = 0.95, ICC = 0.90). Inter-examination variability was 12.7 ± 118.2 mm(3) for AVC and 31.5 ± 219.2 mm(3) for MC. Inter-observer reliability ranged from κ = 0.68 to κ = 0.92 for AVC and from κ = 0.20 to κ = 0.66 for MC. Inter-observer ICC was 0.94 for AVC and ranged from 0.56 to 0.97 for MC. Inter-observer variability ranged from -30.5 ± 252.0 mm(3) to 84.0 ± 240.5 mm(3) for AVC and from -95.2 ± 210.0 mm(3) to 303.7 ± 501.6 mm(3) for MC. CONCLUSIONS: AVC can be quantified with excellent reliability on ungated unenhanced low-dose chest CT, but manual detection of MC can be subject to substantial inter-observer variability. Lung cancer screening CT may be used for detection and quantification of cardiac valve calcifications. KEY POINTS: • Low-dose unenhanced ungated chest computed tomography can detect cardiac valve calcifications. • However, calcified cardiac valves are not reported by most radiologists. • Inter-observer and inter-examination variability of aortic valve calcifications is sufficient for longitudinal studies. • Volumetric measurement variability of mitral valve and annulus calcifications is substantial.

Variation in quantitative CT air trapping in heavy smokers on repeat CT examinations.

OBJECTIVES: To determine the variation in quantitative computed tomography (CT) measures of air trapping in low-dose chest CTs of heavy smokers. METHODS: We analysed 45 subjects from a lung cancer screening trial, examined by CT twice within 3 months. Inspiratory and expiratory low-dose CT was obtained using breath hold instructions. CT air trapping was defined as the percentage of voxels in expiratory CT with an attenuation below -856 HU (EXP(-856)) and the expiratory to inspiratory ratio of mean lung density (E/I-ratio(MLD)). Variation was determined using limits of agreement, defined as 1.96 times the standard deviation of the mean difference. The effect of both lung volume correction and breath hold reproducibility was determined. RESULTS: The limits of agreement for uncorrected CT air trapping measurements were -15.0 to 11.7 % (EXP(-856)) and -9.8 to 8.0 % (E/I-ratio(MLD)). Good breath hold reproducibility significantly narrowed the limits for EXP(-856) (-10.7 to 7.5 %, P = 0.002), but not for E/I-ratio(MLD) (-9.2 to 7.9 %, P = 0.75). Statistical lung volume correction did not improve the limits for EXP(-856) (-12.5 to 8.8 %, P = 0.12) and E/I-ratio(MLD) (-7.5 to 5.8 %, P = 0.17). CONCLUSIONS: Quantitative air trapping measures on low-dose CT of heavy smokers show considerable variation on repeat CT examinations, regardless of lung volume correction or reproducible breath holds. KEY POINTS: Computed tomography quantitatively measures small airways disease in heavy smokers. Measurements of air trapping vary considerably on repeat CT examinations. Variation remains substantial even with reproducible breath holds and lung volume correction.

The effect of iterative reconstruction on computed tomography assessment of emphysema, air trapping and airway dimensions.

OBJECTIVES: To determine the influence of iterative reconstruction (IR) on quantitative computed tomography (CT) measurements of emphysema, air trapping, and airway wall and lumen dimensions, compared to filtered back-projection (FBP). METHODS: Inspiratory and expiratory chest CTs of 75 patients (37 male, 38 female; mean age 64.0 ± 5.7 years) were reconstructed using FBP and IR. CT emphysema, CT air trapping and airway dimensions of a segmental bronchus were quantified using several commonly used quantification methods. The two algorithms were compared using the concordance correlation coefficient (p (c)) and Wilcoxon signed rank test. RESULTS: Only the E/I-ratio(MLD) as a measure of CT air trapping and airway dimensions showed no significant differences between the algorithms, whereas all CT emphysema and the other CT air trapping measures were significantly different at IR when compared to FBP (P < 0.001). CONCLUSION: The evaluated IR algorithm significantly influences quantitative CT measures in the assessment of emphysema and air trapping. However, the E/I-ratio(MLD) as a measure of CT air trapping, as well as the airway measurements, is unaffected by this reconstruction method. Quantitative CT of the lungs should be performed with careful attention to the CT protocol, especially when iterative reconstruction is introduced. KEY POINTS : • New techniques in CT allow numerous quantitative measurements of lung function. • Iterative reconstruction influences quantitative CT measurements of emphysema and air trapping. • Expiratory-to-inspiratory ratio of mean lung density and airway measurements are unaffected by iterative reconstruction. • Quantitative lung-CT should be performed with careful attention to the CT protocol.

Bringing AI to the clinic: blueprint for a vendor-neutral AI deployment infrastructure.

AI provides tremendous opportunities for improving patient care, but at present there is little evidence of real-world uptake. An important barrier is the lack of well-designed, vendor-neutral and future-proof infrastructures for deployment. Because current AI algorithms are very narrow in scope, it is expected that a typical hospital will deploy many algorithms concurrently. Managing stand-alone point solutions for all of these algorithms will be unmanageable. A solution to this problem is a dedicated platform for deployment of AI. Here we describe a blueprint for such a platform and the high-level design and implementation considerations of such a system that can be used clinically as well as for research and development. Close collaboration between radiologists, data scientists, software developers and experts in hospital IT as well as involvement of patients is crucial in order to successfully bring AI to the clinic.

Identifying error types in visual diagnostic skill assessment.

BACKGROUND: Misinterpretation of medical images is an important source of diagnostic error. Errors can occur in different phases of the diagnostic process. Insight in the error types made by learners is crucial for training and giving effective feedback. Most diagnostic skill tests however penalize diagnostic mistakes without an eye for the diagnostic process and the type of error. A radiology test with stepwise reasoning questions was used to distinguish error types in the visual diagnostic process. We evaluated the additional value of a stepwise question-format, in comparison with only diagnostic questions in radiology tests. METHODS: Medical students in a radiology elective (n=109) took a radiology test including 11-13 cases in stepwise question-format: marking an abnormality, describing the abnormality and giving a diagnosis. Errors were coded by two independent researchers as perception, analysis, diagnosis, or undefined. Erroneous cases were further evaluated for the presence of latent errors or partial knowledge. Inter-rater reliabilities and percentages of cases with latent errors and partial knowledge were calculated. RESULTS: The stepwise question-format procedure applied to 1351 cases completed by 109 medical students revealed 828 errors. Mean inter-rater reliability of error type coding was Cohen's κ=0.79. Six hundred and fifty errors (79%) could be coded as perception, analysis or diagnosis errors. The stepwise question-format revealed latent errors in 9% and partial knowledge in 18% of cases. CONCLUSIONS: A stepwise question-format can reliably distinguish error types in the visual diagnostic process, and reveals latent errors and partial knowledge.

Increasing Authenticity of Simulation-Based Assessment in Diagnostic Radiology.

INTRODUCTION: Clinical reasoning in diagnostic imaging professions is a complex skill that requires processing of visual information and image manipulation skills. We developed a digital simulation-based test method to increase authenticity of image interpretation skill assessment. METHODS: A digital application, allowing volumetric image viewing and manipulation, was used for three test administrations of the national Dutch Radiology Progress Test for residents. This study describes the development and implementation process in three phases. To assess authenticity of the digital tests, perceived image quality and correspondence to clinical practice were evaluated and compared with previous paper-based tests (PTs). Quantitative and qualitative evaluation results were used to improve subsequent tests. RESULTS: Authenticity of the first digital test was not rated higher than the PTs. Test characteristics and environmental conditions, such as image manipulation options and ambient lighting, were optimized based on participants' comments. After adjustments in the third digital test, participants favored the image quality and clinical correspondence of the digital image questions over paper-based image questions. CONCLUSIONS: Digital simulations can increase authenticity of diagnostic radiology assessments compared with paper-based testing. However, authenticity does not necessarily increase with higher fidelity. It can be challenging to simulate the image interpretation task of clinical practice in a large-scale assessment setting, because of technological limitations. Optimizing image manipulation options, the level of ambient light, time limits, and question types can help improve authenticity of simulation-based radiology assessments.

Volumetric CT-images improve testing of radiological image interpretation skills.

RATIONALE AND OBJECTIVES: Current radiology practice increasingly involves interpretation of volumetric data sets. In contrast, most radiology tests still contain only 2D images. We introduced a new testing tool that allows for stack viewing of volumetric images in our undergraduate radiology program. We hypothesized that tests with volumetric CT-images enhance test quality, in comparison with traditional completely 2D image-based tests, because they might better reflect required skills for clinical practice. MATERIALS AND METHODS: Two groups of medical students (n=139; n=143), trained with 2D and volumetric CT-images, took a digital radiology test in two versions (A and B), each containing both 2D and volumetric CT-image questions. In a questionnaire, they were asked to comment on the representativeness for clinical practice, difficulty and user-friendliness of the test questions and testing program. Students' test scores and reliabilities, measured with Cronbach's alpha, of 2D and volumetric CT-image tests were compared. RESULTS: Estimated reliabilities (Cronbach's alphas) were higher for volumetric CT-image scores (version A: .51 and version B: .54), than for 2D CT-image scores (version A: .24 and version B: .37). Participants found volumetric CT-image tests more representative of clinical practice, and considered them to be less difficult than volumetric CT-image questions. However, in one version (A), volumetric CT-image scores (M 80.9, SD 14.8) were significantly lower than 2D CT-image scores (M 88.4, SD 10.4) (p<.001). The volumetric CT-image testing program was considered user-friendly. CONCLUSION: This study shows that volumetric image questions can be successfully integrated in students' radiology testing. Results suggests that the inclusion of volumetric CT-images might improve the quality of radiology tests by positively impacting perceived representativeness for clinical practice and increasing reliability of the test.

Support for external validity of radiological anatomy tests using volumetric images.

RATIONALE AND OBJECTIVES: Radiology practice has become increasingly based on volumetric images (VIs), but tests in medical education still mainly involve two-dimensional (2D) images. We created a novel, digital, VI test and hypothesized that scores on this test would better reflect radiological anatomy skills than scores on a traditional 2D image test. To evaluate external validity we correlated VI and 2D image test scores with anatomy cadaver-based test scores. MATERIALS AND METHODS: In 2012, 246 medical students completed one of two comparable versions (A and B) of a digital radiology test, each containing 20 2D image and 20 VI questions. Thirty-three of these participants also took a human cadaver anatomy test. Mean scores and reliabilities of the 2D image and VI subtests were compared and correlated with human cadaver anatomy test scores. Participants received a questionnaire about perceived representativeness and difficulty of the radiology test. RESULTS: Human cadaver test scores were not correlated with 2D image scores, but significantly correlated with VI scores (r = 0.44, P < .05). Cronbach's α reliability was 0.49 (A) and 0.65 (B) for the 2D image subtests and 0.65 (A) and 0.71 (B) for VI subtests. Mean VI scores (74.4%, standard deviation 2.9) were significantly lower than 2D image scores (83.8%, standard deviation 2.4) in version A (P < .001). VI questions were considered more representative of clinical practice and education than 2D image questions and less difficult (both P < .001). CONCLUSIONS: VI tests show higher reliability, a significant correlation with human cadaver test scores, and are considered more representative for clinical practice than tests with 2D images.

Evaluation of automatic neonatal brain segmentation algorithms: the NeoBrainS12 challenge.

A number of algorithms for brain segmentation in preterm born infants have been published, but a reliable comparison of their performance is lacking. The NeoBrainS12 study (http://neobrains12.isi.uu.nl), providing three different image sets of preterm born infants, was set up to provide such a comparison. These sets are (i) axial scans acquired at 40 weeks corrected age, (ii) coronal scans acquired at 30 weeks corrected age and (iii) coronal scans acquired at 40 weeks corrected age. Each of these three sets consists of three T1- and T2-weighted MR images of the brain acquired with a 3T MRI scanner. The task was to segment cortical grey matter, non-myelinated and myelinated white matter, brainstem, basal ganglia and thalami, cerebellum, and cerebrospinal fluid in the ventricles and in the extracerebral space separately. Any team could upload the results and all segmentations were evaluated in the same way. This paper presents the results of eight participating teams. The results demonstrate that the participating methods were able to segment all tissue classes well, except myelinated white matter.

Automated coronary artery calcification scoring in non-gated chest CT: agreement and reliability.

OBJECTIVE: To determine the agreement and reliability of fully automated coronary artery calcium (CAC) scoring in a lung cancer screening population. MATERIALS AND METHODS: 1793 low-dose chest CT scans were analyzed (non-contrast-enhanced, non-gated). To establish the reference standard for CAC, first automated calcium scoring was performed using a preliminary version of a method employing coronary calcium atlas and machine learning approach. Thereafter, each scan was inspected by one of four trained raters. When needed, the raters corrected initially automaticity-identified results. In addition, an independent observer subsequently inspected manually corrected results and discarded scans with gross segmentation errors. Subsequently, fully automatic coronary calcium scoring was performed. Agatston score, CAC volume and number of calcifications were computed. Agreement was determined by calculating proportion of agreement and examining Bland-Altman plots. Reliability was determined by calculating linearly weighted kappa (κ) for Agatston strata and intraclass correlation coefficient (ICC) for continuous values. RESULTS: 44 (2.5%) scans were excluded due to metal artifacts or gross segmentation errors. In the remaining 1749 scans, median Agatston score was 39.6 (P25-P75∶0-345.9), median volume score was 60.4 mm3 (P25-P75∶0-361.4) and median number of calcifications was 2 (P25-P75∶0-4) for the automated scores. The κ demonstrated very good reliability (0.85) for Agatston risk categories between the automated and reference scores. The Bland-Altman plots showed underestimation of calcium score values by automated quantification. Median difference was 2.5 (p25-p75∶0.0-53.2) for Agatston score, 7.6 (p25-p75∶0.0-94.4) for CAC volume and 1 (p25-p75∶0-5) for number of calcifications. The ICC was very good for Agatston score (0.90), very good for calcium volume (0.88) and good for number of calcifications (0.64). DISCUSSION: Fully automated coronary calcium scoring in a lung cancer screening setting is feasible with acceptable reliability and agreement despite an underestimation of the amount of calcium when compared to reference scores.

Automatic segmentation of eight tissue classes in neonatal brain MRI.

PURPOSE: Volumetric measurements of neonatal brain tissues may be used as a biomarker for later neurodevelopmental outcome. We propose an automatic method for probabilistic brain segmentation in neonatal MRIs. MATERIALS AND METHODS: In an IRB-approved study axial T1- and T2-weighted MR images were acquired at term-equivalent age for a preterm cohort of 108 neonates. A method for automatic probabilistic segmentation of the images into eight cerebral tissue classes was developed: cortical and central grey matter, unmyelinated and myelinated white matter, cerebrospinal fluid in the ventricles and in the extra cerebral space, brainstem and cerebellum. Segmentation is based on supervised pixel classification using intensity values and spatial positions of the image voxels. The method was trained and evaluated using leave-one-out experiments on seven images, for which an expert had set a reference standard manually. Subsequently, the method was applied to the remaining 101 scans, and the resulting segmentations were evaluated visually by three experts. Finally, volumes of the eight segmented tissue classes were determined for each patient. RESULTS: The Dice similarity coefficients of the segmented tissue classes, except myelinated white matter, ranged from 0.75 to 0.92. Myelinated white matter was difficult to segment and the achieved Dice coefficient was 0.47. Visual analysis of the results demonstrated accurate segmentations of the eight tissue classes. The probabilistic segmentation method produced volumes that compared favorably with the reference standard. CONCLUSION: The proposed method provides accurate segmentation of neonatal brain MR images into all given tissue classes, except myelinated white matter. This is the one of the first methods that distinguishes cerebrospinal fluid in the ventricles from cerebrospinal fluid in the extracerebral space. This method might be helpful in predicting neurodevelopmental outcome and useful for evaluating neuroprotective clinical trials in neonates.