How to evaluate perfusion imaging in post-treatment glioma: a comparison of three different analysis methods.

INTRODUCTION: Dynamic susceptibility contrast (DSC) perfusion weighted (PW)-MRI can aid in differentiating treatment related abnormalities (TRA) from tumor progression (TP) in post-treatment glioma patients. Common methods, like the 'hot spot', or visual approach suffer from oversimplification and subjectivity. Using perfusion of the complete lesion potentially offers an objective and accurate alternative. This study aims to compare the diagnostic value and assess the subjectivity of these techniques. METHODS: 50 Glioma patients with enhancing lesions post-surgery and chemo-radiotherapy were retrospectively included. Outcome was determined by clinical/radiological follow-up or biopsy. Imaging analysis used the 'hot spot', volume of interest (VOI) and visual approach. Diagnostic accuracy was compared using receiving operator characteristics (ROC) curves for the VOI and 'hot spot' approach, visual assessment was analysed with contingency tables. Inter-operator agreement was determined with Cohens kappa and intra-class coefficient (ICC). RESULTS: 29 Patients suffered from TP, 21 had TRA. The visual assessment showed poor to substantial inter-operator agreement (κ = -0.72 - 0.68). Reliability of the 'hot spot' placement was excellent (ICC = 0.89), while reference placement was variable (ICC = 0.54). The area under the ROC (AUROC) of the mean- and maximum relative cerebral blood volume (rCBV) (VOI-analysis) were 0.82 and 0.72, while the rCBV-ratio ('hot spot' analysis) was 0.69. The VOI-analysis had a more balanced sensitivity and specificity compared to visual assessment. CONCLUSIONS: VOI analysis of DSC PW-MRI data holds greater diagnostic accuracy in single-moment differentiation of TP and TRA than 'hot spot' or visual analysis. This study underlines the subjectivity of visual placement and assessment.

Expertise development in volumetric image interpretation of radiology residents: what do longitudinal scroll data reveal?

The current study used theories on expertise development (the holistic model of image perception and the information reduction hypothesis) as a starting point to identify and explore potentially relevant process measures to monitor and evaluate expertise development in radiology residency training. It is the first to examine expertise development in volumetric image interpretation (i.e., CT scans) within radiology residents using scroll data collected longitudinally over five years of residency training. Consistent with the holistic model of image perception, the percentage of time spent on full runs, i.e. scrolling through more than 50% of the CT-scan slices (global search), decreased within residents over residency training years. Furthermore, the percentage of time spent on question-relevant areas in the CT scans increased within residents over residency training years, consistent with the information reduction hypothesis. Second, we examined if scroll patterns can predict diagnostic accuracy. The percentage of time spent on full runs and the percentage of time spent on question-relevant areas did not predict diagnostic accuracy. Thus, although scroll patterns over training years are consistent with visual expertise theories, they could not be used as predictors of diagnostic accuracy in the current study. Therefore, the relation between scroll patterns and performance needs to be further examined, before process measures can be used to monitor and evaluate expertise development in radiology residency training.

Increasing Authenticity of Simulation-Based Assessment in Diagnostic Radiology.

INTRODUCTION: Clinical reasoning in diagnostic imaging professions is a complex skill that requires processing of visual information and image manipulation skills. We developed a digital simulation-based test method to increase authenticity of image interpretation skill assessment. METHODS: A digital application, allowing volumetric image viewing and manipulation, was used for three test administrations of the national Dutch Radiology Progress Test for residents. This study describes the development and implementation process in three phases. To assess authenticity of the digital tests, perceived image quality and correspondence to clinical practice were evaluated and compared with previous paper-based tests (PTs). Quantitative and qualitative evaluation results were used to improve subsequent tests. RESULTS: Authenticity of the first digital test was not rated higher than the PTs. Test characteristics and environmental conditions, such as image manipulation options and ambient lighting, were optimized based on participants' comments. After adjustments in the third digital test, participants favored the image quality and clinical correspondence of the digital image questions over paper-based image questions. CONCLUSIONS: Digital simulations can increase authenticity of diagnostic radiology assessments compared with paper-based testing. However, authenticity does not necessarily increase with higher fidelity. It can be challenging to simulate the image interpretation task of clinical practice in a large-scale assessment setting, because of technological limitations. Optimizing image manipulation options, the level of ambient light, time limits, and question types can help improve authenticity of simulation-based radiology assessments.

Predictors of Knowledge and Image Interpretation Skill Development in Radiology Residents.

Purpose To investigate knowledge and image interpretation skill development in residency by studying scores on knowledge and image questions on radiology tests, mediated by the training environment. Materials and Methods Ethical approval for the study was obtained from the ethical review board of the Netherlands Association for Medical Education. Longitudinal test data of 577 of 2884 radiology residents who took semiannual progress tests during 5 years were retrospectively analyzed by using a nonlinear mixed-effects model taking training length as input variable. Tests included nonimage and image questions that assessed knowledge and image interpretation skill. Hypothesized predictors were hospital type (academic or nonacademic), training hospital, enrollment age, sex, and test date. Results Scores showed a curvilinear growth during residency. Image scores increased faster during the first 3 years of residency and reached a higher maximum than knowledge scores (55.8% vs 45.1%). The slope of image score development versus knowledge question scores of 1st-year residents was 16.8% versus 12.4%, respectively. Training hospital environment appeared to be an important predictor in both knowledge and image interpretation skill development (maximum score difference between training hospitals was 23.2%; P < .001). Conclusion Expertise developed rapidly in the initial years of radiology residency and leveled off in the 3rd and 4th training year. The shape of the curve was mainly influenced by the specific training hospital. © RSNA, 2017 Online supplemental material is available for this article.

Identifying error types in visual diagnostic skill assessment.

BACKGROUND: Misinterpretation of medical images is an important source of diagnostic error. Errors can occur in different phases of the diagnostic process. Insight in the error types made by learners is crucial for training and giving effective feedback. Most diagnostic skill tests however penalize diagnostic mistakes without an eye for the diagnostic process and the type of error. A radiology test with stepwise reasoning questions was used to distinguish error types in the visual diagnostic process. We evaluated the additional value of a stepwise question-format, in comparison with only diagnostic questions in radiology tests. METHODS: Medical students in a radiology elective (n=109) took a radiology test including 11-13 cases in stepwise question-format: marking an abnormality, describing the abnormality and giving a diagnosis. Errors were coded by two independent researchers as perception, analysis, diagnosis, or undefined. Erroneous cases were further evaluated for the presence of latent errors or partial knowledge. Inter-rater reliabilities and percentages of cases with latent errors and partial knowledge were calculated. RESULTS: The stepwise question-format procedure applied to 1351 cases completed by 109 medical students revealed 828 errors. Mean inter-rater reliability of error type coding was Cohen's κ=0.79. Six hundred and fifty errors (79%) could be coded as perception, analysis or diagnosis errors. The stepwise question-format revealed latent errors in 9% and partial knowledge in 18% of cases. CONCLUSIONS: A stepwise question-format can reliably distinguish error types in the visual diagnostic process, and reveals latent errors and partial knowledge.

Tests, Quizzes, and Self-Assessments: How to Construct a High-Quality Examination.

OBJECTIVE: The purposes of this article are to highlight aspects of tests that increase or decrease their effectiveness and to provide guidelines for constructing high-quality tests in radiology. CONCLUSION: Many radiologists help construct tests for a variety of purposes. Only well-constructed tests can provide reliable and valuable information about the test taker.

The Importance of Human-Computer Interaction in Radiology E-learning.

With the development of cross-sectional imaging techniques and transformation to digital reading of radiological imaging, e-learning might be a promising tool in undergraduate radiology education. In this systematic review of the literature, we evaluate the emergence of image interaction possibilities in radiology e-learning programs and evidence for effects of radiology e-learning on learning outcomes and perspectives of medical students and teachers. A systematic search in PubMed, EMBASE, Cochrane, ERIC, and PsycInfo was performed. Articles were screened by two authors and included when they concerned the evaluation of radiological e-learning tools for undergraduate medical students. Nineteen articles were included. Seven studies evaluated e-learning programs with image interaction possibilities. Students perceived e-learning with image interaction possibilities to be a useful addition to learning with hard copy images and to be effective for learning 3D anatomy. Both e-learning programs with and without image interaction possibilities were found to improve radiological knowledge and skills. In general, students found e-learning programs easy to use, rated image quality high, and found the difficulty level of the courses appropriate. Furthermore, they felt that their knowledge and understanding of radiology improved by using e-learning. In conclusion, the addition of radiology e-learning in undergraduate medical education can improve radiological knowledge and image interpretation skills. Differences between the effect of e-learning with and without image interpretation possibilities on learning outcomes are unknown and should be subject to future research.

Support for external validity of radiological anatomy tests using volumetric images.

RATIONALE AND OBJECTIVES: Radiology practice has become increasingly based on volumetric images (VIs), but tests in medical education still mainly involve two-dimensional (2D) images. We created a novel, digital, VI test and hypothesized that scores on this test would better reflect radiological anatomy skills than scores on a traditional 2D image test. To evaluate external validity we correlated VI and 2D image test scores with anatomy cadaver-based test scores. MATERIALS AND METHODS: In 2012, 246 medical students completed one of two comparable versions (A and B) of a digital radiology test, each containing 20 2D image and 20 VI questions. Thirty-three of these participants also took a human cadaver anatomy test. Mean scores and reliabilities of the 2D image and VI subtests were compared and correlated with human cadaver anatomy test scores. Participants received a questionnaire about perceived representativeness and difficulty of the radiology test. RESULTS: Human cadaver test scores were not correlated with 2D image scores, but significantly correlated with VI scores (r = 0.44, P < .05). Cronbach's α reliability was 0.49 (A) and 0.65 (B) for the 2D image subtests and 0.65 (A) and 0.71 (B) for VI subtests. Mean VI scores (74.4%, standard deviation 2.9) were significantly lower than 2D image scores (83.8%, standard deviation 2.4) in version A (P < .001). VI questions were considered more representative of clinical practice and education than 2D image questions and less difficult (both P < .001). CONCLUSIONS: VI tests show higher reliability, a significant correlation with human cadaver test scores, and are considered more representative for clinical practice than tests with 2D images.

Volumetric and two-dimensional image interpretation show different cognitive processes in learners.

RATIONALE AND OBJECTIVES: In current practice, radiologists interpret digital images, including a substantial amount of volumetric images. We hypothesized that interpretation of a stack of a volumetric data set demands different skills than interpretation of two-dimensional (2D) cross-sectional images. This study aimed to investigate and compare knowledge and skills used for interpretation of volumetric versus 2D images. MATERIALS AND METHODS: Twenty radiology clerks were asked to think out loud while reading four or five volumetric computed tomography (CT) images in stack mode and four or five 2D CT images. Cases were presented in a digital testing program allowing stack viewing of volumetric data sets and changing views and window settings. Thoughts verbalized by the participants were registered and coded by a framework of knowledge and skills concerning three components: perception, analysis, and synthesis. The components were subdivided into 16 discrete knowledge and skill elements. A within-subject analysis was performed to compare cognitive processes during volumetric image readings versus 2D cross-sectional image readings. RESULTS: Most utterances contained knowledge and skills concerning perception (46%). A smaller part involved synthesis (31%) and analysis (23%). More utterances regarded perception in volumetric image interpretation than in 2D image interpretation (Median 48% vs 35%; z = -3.9; P < .001). Synthesis was less prominent in volumetric than in 2D image interpretation (Median 28% vs 42%; z = -3.9; P < .001). No differences were found in analysis utterances. CONCLUSIONS: Cognitive processes in volumetric and 2D cross-sectional image interpretation differ substantially. Volumetric image interpretation draws predominantly on perceptual processes, whereas 2D image interpretation is mainly characterized by synthesis. The results encourage the use of volumetric images for teaching and testing perceptual skills.

Volumetric CT-images improve testing of radiological image interpretation skills.

RATIONALE AND OBJECTIVES: Current radiology practice increasingly involves interpretation of volumetric data sets. In contrast, most radiology tests still contain only 2D images. We introduced a new testing tool that allows for stack viewing of volumetric images in our undergraduate radiology program. We hypothesized that tests with volumetric CT-images enhance test quality, in comparison with traditional completely 2D image-based tests, because they might better reflect required skills for clinical practice. MATERIALS AND METHODS: Two groups of medical students (n=139; n=143), trained with 2D and volumetric CT-images, took a digital radiology test in two versions (A and B), each containing both 2D and volumetric CT-image questions. In a questionnaire, they were asked to comment on the representativeness for clinical practice, difficulty and user-friendliness of the test questions and testing program. Students' test scores and reliabilities, measured with Cronbach's alpha, of 2D and volumetric CT-image tests were compared. RESULTS: Estimated reliabilities (Cronbach's alphas) were higher for volumetric CT-image scores (version A: .51 and version B: .54), than for 2D CT-image scores (version A: .24 and version B: .37). Participants found volumetric CT-image tests more representative of clinical practice, and considered them to be less difficult than volumetric CT-image questions. However, in one version (A), volumetric CT-image scores (M 80.9, SD 14.8) were significantly lower than 2D CT-image scores (M 88.4, SD 10.4) (p<.001). The volumetric CT-image testing program was considered user-friendly. CONCLUSION: This study shows that volumetric image questions can be successfully integrated in students' radiology testing. Results suggests that the inclusion of volumetric CT-images might improve the quality of radiology tests by positively impacting perceived representativeness for clinical practice and increasing reliability of the test.

Computed tomography pulmonary angiography in acute pulmonary embolism: the effect of a computer-assisted detection prototype used as a concurrent reader.

PURPOSE: To assess the effect of computer-assisted detection (CAD) on diagnostic accuracy, reader confidence, and reading time when used as a concurrent reader for the detection of acute pulmonary embolism in computed tomography pulmonary angiography. MATERIALS AND METHODS: In this institutional review board-approved retrospective study, 6 observers with varying experience evaluated 158 negative and 38 positive consecutive computed tomography pulmonary angiographies (mean patient age 60 y; 115 women) without and with CAD as a concurrent reader. Readers were asked to determine the presence of pulmonary embolism, assess their diagnostic confidence using a 5-point scale, and document their reading time. Results were compared with an independent standard established by 2 readers, and a third chest radiologist was consulted in case of discordant findings. RESULTS: Using logistic regression for repeated measurements, we found a significant increase in readers' sensitivity (P<0.001) without loss of specificity (P=0.855) with the effects being reader dependent (P<0.001). Sensitivities varied from 68% to 100% without CAD and from 76% to 100% with CAD. A 2-way analysis of variance showed a small but significant decrease in reading time (P<0.001), with the duration varying between 24 and 208 seconds without CAD and between 17 and 196 seconds with CAD, and a significant increase in readers' confidence scores using CAD as a concurrent reader (P<0.001). CONCLUSIONS: CAD as a concurrent reader has the potential to increase readers' sensitivity and confidence with a decrease in reading time without loss of specificity. The differences between readers, however, require further evaluation of CAD as a concurrent reader in a larger trial before stronger conclusions can be drawn.

Construct validation of progress testing to measure knowledge and visual skills in radiology.

BACKGROUND: The Dutch Radiology Progress Test (DRPT) monitors the acquisition of knowledge and visual skills of radiology residents in the Netherlands. AIM: We aimed to evaluate the quality of progress testing in postgraduate radiology training by studying the reliability of the DRPT and finding an indication for its construct validity. We expected that knowledge would increase rapidly in the first years of residency, leveling-off in later years, to allow for the development of visual skills. We hypothesized that scores on the DRPT reflect this pattern. METHODS: Internal consistencies were estimated with Cronbach's alpha. Performance increase over program years were tested using one-way analysis of variance. RESULTS: Data were available for 498 residents (2281 test results). Reliabilities were around Cronbach's alpha 0.90. There was a significant difference in the mean test results between the first three years of residency. After the fourth year no significant increase in test scores on knowledge could be measured on eight tests. The same pattern occurred for scores on visual skills. However, visual skills scores tend to increase more sharply than knowledge scores. CONCLUSION: We found support for the reliability and construct validity of the DRPT. However, assessment on visual skill development needs further exploration.