Full Resolution Simulation for Evaluation of Critical Care Imaging Interpretation; Part 2: Random Effects Reveal the Interplay Between Case Difficulty, Resident Competence, and the Training Environment.

RATIONALE AND OBJECTIVES: To further characterize empirical data from a full-resolution simulation of critical care imaging coupled with post hoc grading of resident's interpretations by senior radiologists. To present results from estimating the random effects terms in a comprehensive mixed (hierarchical) regression model. MATERIALS AND METHODS: After accounting for 9 fixed effects detailed in Part 1 of this paper, we estimated normally distributed random effects, expressed in terms of score offsets for each case, resident, program, and grader. RESULTS: The fixed effects alone explained 8.8% of score variation and adding the random effects increased explanatory power of the model to account for 36% of score variation. As quantified by intraclass correlation coefficient (ICC = 28.5%; CI: 25.1-31.6) the majority of score variation is directly attributable to the case at hand. This "case difficulty" measure has reliability of 95%. Individual residents accounted for much of the remaining score variation (ICC = 5.3%; CI: 4.6-5.9) after adjusting for all other effects including level of training. The reliability of this "resident competence" measure is 82%. Residency training program influence on scores was small (ICC = 1.1%; CI: 0.42-1.7). Although a few significantly high and low ones can be identified, reliability of 73% militates for caution. At the same time, low intraprogram variation is very encouraging. Variation attributable to differences between graders was minimal (ICC = 0.58%; CI: 0.0-1.2) which reassures us that the method of scoring is reliable, consistent, and likely extensible. CONCLUSION: Full resolution simulation based evaluation of critical care radiology interpretation is being conducted remotely and efficiently at large scale. A comprehensive mixed model of the resulting scores reliably quantifies case difficulty and resident competence.

Full Resolution Simulation for Evaluation of Critical Care Imaging Interpretation; Part 1: Fixed Effects Identify Influences of Exam, Specialty, Fatigue, and Training on Resident Performance.

RATIONALE AND OBJECTIVES: To describe our full-resolution simulation of critical care imaging coupled with posthoc grading of resident's interpretations and present results from the fixed effects terms in a comprehensive mixed regression model of the resulting scores. MATERIALS AND METHODS: The system delivered full resolution DICOM studies via clinical-grade viewing software integrated with a custom built web-based workflow and reporting system. The interpretations submitted by participating residents from 47 different programs were graded (scores of 0-10) on a case by case basis by a cadre of faculty members from our department. The data from 5 yearly (2014-2018) cycles consisting of 992 separate 65 case, 8 hour simulation sessions were collated from the transaction records. We used a mixed (hierarchical) statistical model with nine fixed and four random independent variables. In this paper, we present the results from the nine fixed effects. RESULTS: There were 19,916/63,839 (27.0%, CI 26.7%-27.4%) scores in the 0-2 range (i.e., clinically significant miss). Neurological cases were more difficult with adjusted scores 2.3 (CI 1.9-3.2) lower than body/musculoskeletal cases. There was a small (0.3, CI 0.20-0.38 points) but highly significant (p<0.0001) decrease in score for the final 13/65 cases (fifth quintile) as evidence of fatigue during the last hour of an 8 hour shift. By comparing adjusted scores from mid-R1 (quarter 3) to late-R3 (quarter 12) we estimate the training effect as an increase of 2.2 (CI 1.90-2.50) points. CONCLUSION: Full resolution simulation based evaluation of critical care radiology interpretation is being conducted remotely and efficiently at large scale. Analysis of the resulting scores yields multiple insights into the interpretative process.

Behavioral Health Factors as Predictors of Emergency Department Use in the High-Risk, High-Cost Medicare Population.

OBJECTIVE: This study measured the presence, extent, and type of behavioral health factors in a high-cost Medicare population and their association with the probability and intensity of emergency department (ED) use. METHODS: Retrospective claims analysis and a comprehensive electronic medical record-based review were conducted for patients enrolled in a 65-month prospective care management program at an academic tertiary medical center (N=3,620). A two-part model used multivariable logistic regression to evaluate the effect of behavioral health factors on the probability of ED use, complemented by a Poisson model to measure the number of ED visits. Control variables included demographic characteristics, poststudy survival, and hierarchical condition category risk score. RESULTS: After analyses controlled for comorbidities and other relevant variables, patients with two or more behavioral health diagnosis categories or two or more behavioral health medications were about twice as likely as those without such categories or medications to use the ED. Patients with a diagnosis category of psychosis, neuropsychiatric disorders, sleep disorders, or adjustment disorders were significantly more likely than those without these disorders to use the ED. Most primary ED diagnoses were not of behavioral health conditions. CONCLUSIONS: Behavioral health factors had a substantial and significant effect on the likelihood and number of ED visits in a population of high-cost Medicare patients. Attention to behavioral health factors as independent predictors of ED use may be useful in influencing ED use in high-cost populations.

Angiosomal radiopathologic analysis of transarterial radioembolization for the treatment of hepatocellular carcinoma.

PURPOSE: To assess the radiopathologic correlation following Yttrium-90 transarterial radioembolization (TARE) of hepatocellular carcinoma (HCC) using variable radiodosimetry to identify imaging surrogates of histologic response. METHODS: Twelve patients with HCC underwent ablative (≥ 190 Gy) and/or non-ablative (< 190 Gy) TARE delivered in a segmental, lobar, or combined fashion as a surgical neoadjuvant or bridge to transplantation. Both targeted tumor and treatment angiosome were analyzed before and after TARE utilizing hepatocyte-specific contrast-enhanced MRI or contrast-enhanced CT. Responses were graded using EASL and mRECIST criteria. Histologic findings including percent tumor necrosis and adjacent hepatic substrate effects were correlated with imaging features. RESULTS: Complete pathologic necrosis (CPN) was observed in 7/12 tumors post-TARE. Ablative and non-ablative dosing resulted in CPN in 5/6 and 2/6 tumors, respectively. Hyperintensity on T2-weighted imaging, the absence of hepatocyte-specific gadolinium contrast uptake, and plateau or persistent enhancement kinetics in the angiosome correlated with CPN and performed similarly to EASL and mRECIST criteria in predicting CPN. CONCLUSIONS: The absence of hepatocyte-specific contrast uptake, increased signal on T2-weighted sequences, and plateau or persistent enhancement in the angiosome may represent MRI surrogates of CPN following TARE of HCC. These findings correlated with EASL and mRECIST response criteria. Further investigation is needed to determine the role of these findings as possible adjuncts to conventional imaging criteria.

Images of imaging: how to process and display imaging utilization for large populations.

OBJECTIVE: We propose a method of processing and displaying imaging utilization data for large populations. CONCLUSION: The comprehensive and finely grained picture of imaging utilization yielded by our methods is a first step toward population-based imaging utilization management. We believe that our methods for the categorization and display of imaging utilization will prove to be widely useful.

Provider Feedback about Imaging Appropriateness by Using Scores from Order Entry Decision Support: Raw Rates Misclassify Outliers.

PURPOSE: To determine the relevant physician- and practice-related factors that jointly affect the rate of low-utility imaging examinations (score of 1-3 out of 9) ordered by means of an order entry system that provides normative appropriateness feedback. MATERIALS AND METHODS: This HIPAA-compliant study was approved by the institutional review board under an expedited protocol for analyzing anonymous aggregated administrative data. This is a retrospective study of approximately 250 000 consecutive scheduled outpatient advanced imaging examinations (computed tomography, magnetic resonance imaging, nuclear medicine) ordered by 164 primary care and 379 medical specialty physicians from 2008 to 2012. A hierarchical logistic regression model was used to identify multiple predictors of the probability that an examination received a low utility score. Physician- and practice-specific random effects were estimated to articulate (odds ratio) and quantify (intraclass correlation) interphysician variation. RESULTS: Fixed effects found to be statistically significant predictors of low-utility imaging included examination type, whether the examination was cancelled, status of the person entering the order, and the total number of examinations ordered by the clinician. Neither patient age nor sex had any effect, and there were no secular trends (year of study). The remaining amount of interphysician variation was moderate (intraclass correlation, 22%), whereas the variation between medical specialties and primary care practices was low (intraclass correlation, 5%). The estimated physician-specific effects had reliability of 70%, which makes them just suitable for identifying outliers. CONCLUSION: The authors found that 22% of the variation in the rate of low-utility examinations is attributable to ordering providers and 5% to their specialty or clinic.

Use of imaging in the emergency department: physicians have limited effect on variation.

PURPOSE: To quantify interphysician variation in imaging use during emergency department (ED) visits and examine the contribution of factors to this variation at the patient, visit, and physician level. MATERIALS AND METHODS: This study was HIPAA compliant and approved by the institutional review board of Partners Healthcare System (Boston, Mass), with waiver of informed consent. In this retrospective study of 88 851 consecutive ED visits during 2011 at a large urban teaching hospital, a hierarchical logistic regression model was used to identify multiple predictors for the probability that low- or high-cost imaging would be ordered during a given visit. Physician-specific random effects were estimated to articulate (by odds ratio) and quantify (by intraclass correlation coefficient [ICC]) interphysician variation. RESULTS: Patient- and visit-level factors found to be statistically significant predictors of imaging use included measures of ED busyness, prior ED visit, referral source to the ED, and ED arrival mode. Physician-level factors (eg, sex, years since graduation, annual workload, and residency training) did not correlate with imaging use. The remaining amount of interphysician variation was very low (ICC, 0.97% for low-cost imaging; ICC, 1.07% for high-cost imaging). These physician-specific odds ratios of imaging estimates were moderately reliable at 0.78 (95% confidence interval [CI]: 0.77, 0.79) for low-cost imaging and 0.76 (95% CI: 0.74, 0.78) for high-cost imaging. CONCLUSION: After careful and comprehensive case-mix adjustment by using hierarchical logistic regression, only about 1% of the variability in ED imaging utilization was attributable to physicians.

Email notification combined with off site signing substantially reduces resident approval to faculty verification time.

RATIONALE AND OBJECTIVES: Attending radiologist signature time (AST) is a variable and modifiable component of overall report turnaround time. Delays in finalized reports have potential to undermine radiologists' value as consultants and adversely affect patient care. This study was performed to evaluate the impact of notebook computer distribution and daily automated e-mail notification on reducing AST. MATERIALS AND METHODS: Two simultaneous interventions were initiated in the authors' radiology department in February 2010. These included the distribution of a notebook computer with preloaded software for each attending radiologist to sign radiology reports and daily automated e-mail notifications for unsigned reports. The digital dictation system archive and the radiology information system were queried for all radiology reports produced from January 2009 through August 2010. The time between resident approval and attending radiologist signature before and after the intervention was analyzed. Potential unintended "side effects" of the intervention were also studied. RESULTS: Resident-authored reports were signed, on average, 2.53 hours sooner after the intervention. This represented a highly significant (P = .003) decrease in AST with all else held equal. Postintervention reports were authored by residents at the same rate (about 70%). An unintended "side effect" was that attending radiologists were less likely to make changes to resident-authored reports after the intervention. CONCLUSIONS: E-mail notification combined with offsite signing can reduce AST substantially. Notebook computers with preloaded software streamline the process of accessing, editing, and signing reports. The observed decrease in AST reflects a positive change in the timeliness of report signature.

Free text versus structured format: information transfer efficiency of radiology reports.

OBJECTIVE: We discuss the effect of radiology report format on the accuracy and speed with which reviewers can extract case-specific information. MATERIALS AND METHODS: A Web-based testing mechanism was used to present radiology reports to each of 16 senior medical students and record their answers to 10 multiple choice questions about specific medical content for each of 12 cases. Subjects were randomly assigned to view the reports in either free text or structured format. In addition to number of answers correct for each case, we recorded the time taken for each case and an efficiency score (correctly answered questions per minute). These three outcomes were tested for differences on report format using multifactorial analysis of variance. A postexperimental questionnaire and a mediated focus group elicited subject preference as to radiology report format. RESULTS: There were no significant differences in the three outcomes (score, time, and efficiency) between the free text and structured format conditions. The power of the experiment was sufficient to detect small differences in these outcomes by format. Subjects strongly and consistently expressed a preference for the structured version. CONCLUSION: We assert that free text and itemized (structured) forms of radiology reports are equally efficient and accurate for transmitting case-specific interpretative content to reviewers of the document.

Conceptual approach for the design of radiology reporting interfaces: the talking template.

Within the coming decade, traditional dictation supported by human transcription for radiology reports will be replaced by one or more computerized methods. This paper discusses the cognitive and process efficiency problems arising from currently available technology including speech recognition and menu-driven interfaces. A specific concept for interaction with the reporting interface is proposed. This is called the "talking template" and departs from other designs by providing for all interactions to be mediated through audible prompts and microphone controls. The radiologist can recapture efficiency and cognitive focus by dictating while viewing images without the "look away" problem inherent in other interfaces.

A framework for improving radiology reporting.

The interpretative reports rendered by radiologists are the only tangible manifestation of their expertise, training, and experience. These documents are very often the primary means by which radiologists provide patient care. Radiology reports are extremely variable in form, content, and quality. The authors propose a framework for conceptualizing the reporting process and how it might be improved. This consists of standard language, a structured format, and consistent content. These attributes will be realized by modifying the clinical reporting process, including the creation, storage, transmission, and review of interpretative documents. The authors also point out that changes in training and evaluation must be a part of the process, because they are complementary to purely technical solutions.

Relational data model for the American College of Radiology Appropriateness Criteria.

This article describes a data model for encoding the American College of Radiology Appropriateness Criteria (ACRAC) for selection of diagnostic imaging procedures. These guidelines are recognized widely as an authoritative repository of "best evidence" concerning appropriate radiology tests for a large number of clinical conditions. In its current text document format, the ACRAC is of limited utility for electronic use. The data model the authors propose completely encodes all attributes and domains of the published guidelines and is suitable for translation into any industry standard relational database system. Additionally, the authors have added mappings onto commonly used procedure (CPT) and clinical problem (ICD) coding systems. When populated with the current ACRAC content, such a database could serve as the "master" repository of the guidelines with changes and additions made via an interface built with standard database application development tools. The database also could be made available for incorporation into existing information systems used for order entry, decision support, compliance tracking, and health services research at regional and national levels.