Automated Test-Item Generation System for Retrieval Practice in Radiology Education.

OBJECTIVE: To develop and disseminate an automated item generation (AIG) system for retrieval practice (self-testing) in radiology and to obtain trainee feedback on its educational utility. MATERIALS AND METHODS: An AIG software program (Radmatic) that is capable of generating large numbers of distinct multiple-choice self-testing items from a given "item-model" was created. Instead of writing multiple individual self-testing items, an educator creates an "item-model" for one of two distinct item styles: true/false knowledge based items and image-based items. The software program then uses the item model to generate self-testing items upon trainee request. This internet-based system was made available to all radiology residents at our institution in conjunction with our didactic conferences. After obtaining institutional review board approval and informed consent, a written survey was conducted to obtain trainee feedback. RESULTS: Two faculty members with no computer programming experience were able to create item-models using a standard template. Twenty five of 54 (46%) radiology residents at our institution participated in the study. Twelve of these 25 (48%) study participants reported using the self-testing items regularly, which correlated well with the anonymous website usage statistics. The residents' overall impression and satisfaction with the self-testing items was quite positive, with a score of 7.89 ± 1.91 (mean ± SD) out of 10. Lack of time and email overload were the main reasons provided by residents for not using self-testing items. CONCLUSION: AIG enabled self-testing is technically feasible, and is perceived positively by radiology residents as useful to their education.

The Role of Report Comparison, Analysis, and Discrepancy Categorization in Resident Education.

OBJECTIVE: The purpose of this study was to show the value of automated radiology report comparison and analysis in resident education by providing qualitative and quantitative feedback on the discrepancies between preliminary and finalized reports. MATERIALS AND METHODS: Anonymous surveys on dictation practices and the process of reviewing reports were completed by consenting radiology residents and faculty. All 277 reports obtained across all modalities during the 4-week study were retrieved from the dictation server in both their preliminary and finalized states, for a total of 544 reports. Disparities between these reports were automatically compared side by side and were categorized according to clinical relevance, report quality, or report structure. The frequency of report corrections was compared between junior (postgraduate years [PGYs] 2 and 3) and senior (PGYs 4 and 5) residents. Residents were surveyed regarding the usefulness of the feedback. RESULTS: Eighty-six reports (31%) were verified as unchanged, with no statistically significant difference noted between junior and senior residents (33.2% and 25.9%, respectively; p = 0.03). Of the 370 discrepancies noted in the 191 edited reports, 81 (21.9%) were discrepancies in clinically relevant findings; 106 (28.6%) were discrepancies in report quality; and 183 (49.5%) were discrepancies in report structure, syntax, or both. Although senior residents had a lower rate of discrepancies in the clinical relevance category than did junior residents (12.8% and 26.5%; p = 0.004), they had a higher rate of discrepancies in the report quality category (58.4% and 44.9%; p = 0.02). Surveys of both residents and faculty showed strong support for the project. CONCLUSION: Categorization of corrections was deemed useful by residents and can be helpful in assessing elements of reporting accuracy for individual feedback. Quantitative report comparison and analysis show promise in tailoring resident education at the programmatic level as cumulative data are gathered and trends are analyzed.