The multi-angle extended three-dimensional activities (META) stimulus set: A tool for studying event cognition.

To study complex human activity and how it is perceived and remembered, it is valuable to have large-scale, well-characterized stimuli that are representative of such activity. We present the Multi-angle Extended Three-dimensional Activities (META) stimulus set, a structured and highly instrumented set of extended event sequences performed in naturalistic settings. Performances were captured with two color cameras and a Kinect v2 camera with color and depth sensors, allowing the extraction of three-dimensional skeletal joint positions. We tracked the positions and identities of objects for all chapters using a mixture of manual coding and an automated tracking pipeline, and hand-annotated the timings of high-level actions. We also performed an online experiment to collect normative event boundaries for all chapters at a coarse and fine grain of segmentation, which allowed us to quantify event durations and agreement across participants. We share these materials publicly to advance new discoveries in the study of complex naturalistic activity.

Impact of Follow-Up Imaging Recommendation Specificity on Adherence.

Radiology reports often contain follow-up imaging recommendations, but failure to comply with them in a timely manner can lead to delayed treatment, poor patient outcomes, complications, and legal liability. Using a dataset containing 2,972,164 exams for over 7 years, in this study we explored the association between recommendation specificity on follow-up rates. Our results suggest that explicitly mentioning the follow-up interval as part of a follow-up imaging recommendation has a significant impact on adherence making these recommendations 3 times more likely (95% CI: 2.95 - 3.05) to be followed-up, while explicit mentioning of the follow-up modality did not have a significant impact. Our findings can be incorporated into routine dictation macros so that the follow-up duration is explicitly mentioned whenever clinically applicable, and/or used as the basis for a quality improvement project focussed on improving adherence to follow-up imaging recommendations.

Correlating the Radiological Assessment of Patient Motion with the Incidence of Repeat Sequences Documented by Log Files.

OBJECTIVE: To correlate a radiological assessment of MR motion artifacts with the incidence of repeated sequences and delays derived from modality log files (MLFs) and investigate the suitability of log files for quantifying the operational impact of patient motion. MATERIALS AND METHODS: An experienced, blinded neuroradiologist retrospectively evaluated one full calendar month of sequentially obtained clinical MR exams of the head and/or brain for the presence of motion artifacts using a previously defined clinical grading scale. MLF data were analyzed to extract the occurrence of repeated sequences during the examinations. Statistical analysis included the determination of 95% confidence intervals for repetition ratios, and Welch's t-test to exclude the hypothesis of equal means for different groups of sequences. RESULTS: A total of 213 examinations were evaluated, comprising 1681 MLF-documented sequences, from which 1580 were archived. Radiological motion assessment scores (0, none to 4, severe) were assigned to each archived sequence. Higher motion scores correlated with a higher MLF-derived repetition probability, reflected by the average motion scores assigned to sequences that would be repeated (group 1, mean=2.5), those that are a repeat (group 2, mean=1.9), and those that are not repeated (group 3, mean=1.1) within an exam. The hypothesis of equal means was rejected with P = 5.9 × 10-5 for groups 1 and 2, P = 9.39 × 10-16 for groups 1 and 3, and P = 1.55 × 10-12 for groups 2 and 3. The repetition probability and associated time loss could be quantified for individual sequence types. The total time loss due to repeat sequence acquisition derived from MLFs was greater than four hours. CONCLUSION: Log file data may help assess patterns of scanner and exam performance and may be useful in identifying pitfalls to diagnostic imaging in a clinical environment, particularly with respect to patient motion.

Is There a Difference Between LI-RADS 3 to LI-RADS 5 Progression Assessment Using CT Versus MR? A Retrospective, Single-Center, Longitudinal Study of Patients Who Underwent 5082 Radiologic Examinations for Surveillance of Hepatocellular Carcinoma Over a 43-Month Period.

OBJECTIVE: The Liver Imaging Reporting and Data System (LI-RADS) has been widely applied to CT and MR liver observations in patients at high-risk for hepatocellular carcinoma (HCC). We investigated the impact of CT vs MR in upgrading LI-RADS 3 to LI-RADS 5 observations using a large cohort of high-risk patients. METHODS: We performed a retrospective, longitudinal study of CT and MR radiographic reports (June 2013 - February 2017) with an assigned LI-RADS category. A final population of 757 individual scans and 212 high-risk patients had at least one LI-RADS 3 observation. Differences in observation time to progression between modalities were determined using uni- and multivariable analysis. RESULTS: Of the 212 patients with a LI-RADS 3 observation, 52 (25%) had progression to LI-RADS 5. Tp ranged from 64 - 818 days (median: 196 days). One hundred and three patients (49%) had MR and 109 patients (51%) had CT as their index study. Twenty-four patients with an MR index exam progressed to LI-RADS 5 during the follow-up interval, with progression rates of 22% (CI:13%-30%) at 1 year and 29% (CI:17%-40%) at 2 years. Twenty-eight patients with a CT index exam progressed to LI-RADS 5 during follow-up, with progression rates of 26% (CI:16%-35%) at 1 year and 31% (CI:19%-41%) at 2 years. Progression rates were not significantly different between patients whose LI-RADS 3 observation was initially diagnosed on MR vs CT (HR: 0.81, P = 0.44). DISCUSSION: MR and CT modalities are comparable for demonstrating progression from LI-RADS 3 to 5 for high risk patients.

Stochastic programming for outpatient scheduling with flexible inpatient exam accommodation.

This study is concerned with the determination of an optimal appointment schedule in an outpatient-inpatient hospital system where the inpatient exams can be cancelled based on certain rules while the outpatient exams cannot be cancelled. Stochastic programming models were formulated and solved to tackle the stochasticity in the procedure durations and patient arrival patterns. The first model, a two-stage stochastic programming model, is formulated to optimize the slot size. The second model further optimizes the inpatient block (IPB) placement and slot size simultaneously. A computational method is developed to solve the second optimization problem. A case study is conducted using the data from Magnetic Resonance Imaging (MRI) centers of Lahey Hospital and Medical Center (LHMC). The current schedule and the schedules obtained from the optimization models are evaluated and compared using simulation based on FlexSim Healthcare. Results indicate that the overall weighted cost can be reduced by 11.6% by optimizing the slot size and can be further reduced by an additional 12.6% by optimizing slot size and IPB placement simultaneously. Three commonly used sequencing rules (IPBEG, OPBEG, and a variant of ALTER rule) were also evaluated. The results showed that when optimization tools are not available, ALTER variant which evenly distributes the IPBs across the day has the best performance. Sensitivity analysis of weights for patient waiting time, machine idle time and exam cancellations further supports the superiority of ALTER variant sequencing rules compared to the other sequencing methods. A Pareto frontier was also developed and presented between patient waiting time and machine idle time to enable medical centers with different priorities to obtain solutions that accurately reflect their respective optimal tradeoffs. An extended optimization model was also developed to incorporate the emergency patient arrivals. The optimal schedules from the extended model show only minor differences compared to those from the original model, thus proving the robustness of the scheduling solutions obtained from our optimal models against the impacts of emergency patient arrivals.

HIGHLIGHTS: Timestamped operational data was analyzed to identify sources of uncertainty and delays. Stochastic programming models were developed to optimize slot size and inpatient block placement. A case study showed that the optimized schedules can reduce overall costs by 23%. Distributing inpatient and outpatient slots evenly throughout the day provides the best performance. A Pareto frontier was developed to allow practitioners to choose their own best tradeoffs between multiple objectives.

Framework for Extracting Critical Findings in Radiology Reports.

Critical results reporting guidelines demand that certain critical findings are communicated to the responsible provider within a specific period of time. In this paper, we discuss a generic report processing pipeline to extract critical findings within the dictated report to allow for automation of quality and compliance oversight using a production dataset containing 1,210,858 radiology exams. Algorithm accuracy on an annotated dataset having 327 sentences was 91.4% (95% CI 87.6-94.2%). Our results show that most critical findings are diagnosed on CT and MR exams and that intracranial hemorrhage and fluid collection are the most prevalent at our institution. 1.6% of the exams were found to have at least one of the ten critical findings we focused on. This methodology can enable detailed analysis of critical results reporting for research, workflow management, compliance, and quality assurance.

Determining Follow-Up Imaging Study Using Radiology Reports.

Radiology reports often contain follow-up imaging recommendations. Failure to comply with these recommendations in a timely manner can lead to delayed treatment, poor patient outcomes, complications, unnecessary testing, lost revenue, and legal liability. The objective of this study was to develop a scalable approach to automatically identify the completion of a follow-up imaging study recommended by a radiologist in a preceding report. We selected imaging-reports containing 559 follow-up imaging recommendations and all subsequent reports from a multi-hospital academic practice. Three radiologists identified appropriate follow-up examinations among the subsequent reports for the same patient, if any, to establish a ground-truth dataset. We then trained an Extremely Randomized Trees that uses recommendation attributes, study meta-data and text similarity of the radiology reports to determine the most likely follow-up examination for a preceding recommendation. Pairwise inter-annotator F-score ranged from 0.853 to 0.868; the corresponding F-score of the classifier in identifying follow-up exams was 0.807. Our study describes a methodology to automatically determine the most likely follow-up exam after a follow-up imaging recommendation. The accuracy of the algorithm suggests that automated methods can be integrated into a follow-up management application to improve adherence to follow-up imaging recommendations. Radiology administrators could use such a system to monitor follow-up compliance rates and proactively send reminders to primary care providers and/or patients to improve adherence.

Environmental Factors Predictive of No-Show Visits in Radiology: Observations of Three Million Outpatient Imaging Visits Over 16 Years.

PURPOSE: To evaluate the impact of environmental and socioeconomic factors on outpatient cancellations and "no-show visits" (NSVs) in radiology. MATERIALS AND METHODS: We conducted a retrospective analysis by collecting environmental factor data related to outpatient radiology visits occurring between 2000 and 2015 at our multihospital academic institution. Appointment attendance records were joined with daily weather observations from the National Oceanic and Atmospheric Administration and estimated median income from the US Census American Community Survey. A multivariate logistic regression model was built to examine relationships between NSV rate and median income, commute distance, maximum daily temperature, and daily snowfall. RESULTS: There were 270,574 (8.0%) cancellations and 87,407 (2.6%) NSVs among 3,379,947 scheduled outpatient radiology appointments and 575,206 unique patients from 2000 to 2015. Overall cancellation rates decreased from 14% to 8%, and NSV rates decreased from 6% to 1% as median income increased from $20,000 to $120,000 per year. In a multivariate model, the odds of NSV decreased 10.7% per $10,000 increase in median income (95% confidence interval [CI]: 10.3%-11.1%) and 2.0% per 10°F increase in maximum daily temperature (95% CI: 1.3%-1.6%). The odds of NSV increased 1.4% per 10-mile increase in commute distance (95% CI: 1.3%-1.6%) and 4.5% per 1-inch increase in daily snowfall (95% CI: 3.6%-5.3%). Commute distance was more strongly associated with NSV for those in the two lower tertiles of income than the highest tertile (P < .001). CONCLUSION: Environmental factors are strongly associated with patients' attendance at scheduled outpatient radiology examinations. Modeling of appointment failure risk based on environmental features can help increase the attendance of outpatient radiology appointments.

Automated Tracking of Follow-Up Imaging Recommendations.

OBJECTIVE. Radiology reports often contain follow-up imaging recommendations. Failure to comply with these recommendations in a timely manner can lead to poor patient outcomes, complications, and legal liability. As such, the primary objective of this research was to determine adherence rates to follow-up recommendations. MATERIALS AND METHODS. Radiology-related examination data, including report text, for examinations performed between June 1, 2015, and July 31, 2017, were extracted from the radiology departments at the University of Washington (UW) and Lahey Hospital and Medical Center (LHMC). The UW dataset contained 923,885 examinations, and the LHMC dataset contained 763,059 examinations. A 1-year period was used for detection of imaging recommendations and up to 14-months for the follow-up examination to be performed. RESULTS. On the basis of an algorithm with 97.9% detection accuracy, the follow-up imaging recommendation rate was 11.4% at UW and 20.9% at LHMC. Excluding mammography examinations, the overall follow-up imaging adherence rate was 51.9% at UW (range, 44.4% for nuclear medicine to 63.0% for MRI) and 52.0% at LHMC (range, 30.1% for fluoroscopy to 63.2% for ultrasound) using a matcher algorithm with 76.5% accuracy. CONCLUSION. This study suggests that follow-up imaging adherence rates vary by modality and between sites. Adherence rates can be influenced by various legitimate factors. Having the capability to identify patients who can benefit from patient engagement initiatives is important to improve overall adherence rates. Monitoring of follow-up adherence rates over time and critical evaluation of variation in recommendation patterns across the practice can inform measures to standardize and help mitigate risk.

Probabilistic Modeling of Exam Durations in Radiology Procedures.

In this paper, we model the statistical properties of imaging exam durations using parametric probability distributions such as the Gaussian, Gamma, Weibull, lognormal, and log-logistic. We establish that in a majority of radiology procedures, the underlying distribution of exam durations is best modeled by a log-logistic distribution, while the Gaussian has the poorest fit among the candidates. Further, through illustrative examples, we show how business insights and workflow analytics can be significantly impacted by making the correct (log-logistic) versus incorrect (Gaussian) model choices.

Understanding Why Patients No-Show: Observations of 2.9 Million Outpatient Imaging Visits Over 16 Years.

PURPOSE: To understand why patients "no-show" for imaging appointments, and to provide new insights for improving resource utilization. MATERIALS AND METHODS: We conducted a retrospective analysis of nearly 2.9 million outpatient examinations in our radiology information system from 2000 to 2015 at our multihospital academic institution. No-show visits were identified by the "reason code" entry "NOSHOW" in our radiology information system. We restricted data to radiography, CT, mammography, MRI, ultrasound, and nuclear medicine examinations that included all studied variables. These variables included modality, patient age, appointment time, day of week, and scheduling lead time. Multivariate logistic regression was used to identify factors associated with no-show visits. RESULTS: Out of 2,893,626 patient visits that met our inclusion criteria, there were 94,096 no-shows during the 16-year period. Rates of no-show visits varied from 3.36% in 2000 to 2.26% in 2015. The effect size for no-shows was strongest for modality and scheduling lead time. Mammography had the highest modality no-show visit rate of 6.99% (odds ratio [OR] 5.38, P < .001) compared with the lowest modality rate of 1.25% in radiography. Scheduling lead time greater than 6 months was associated with more no-show visits than scheduling within 1 week (OR 3.18, P < .001). Patients 60 years and older were less likely to miss imaging appointments than patients under 40 (OR 0.70, P < .001). Mondays and Saturdays had significantly higher rates of no-show than Sundays (OR 1.52 and 1.51, P < .001). CONCLUSION: Modality type and scheduling lead time were the most predictive factors of no-show. This may be used to guide new interventions such as targeted reminders and flexible scheduling.

Determining Adherence to Follow-up Imaging Recommendations.

PURPOSE: Radiology reports often contain follow-up imaging recommendations. However, these recommendations are not always followed up by referring physicians and patients. Failure to comply in a timely manner can lead to delayed treatment, poor patient outcomes, unnecessary testing, lost revenue, and legal liability. Therefore, the primary objective of this research was to determine adherence rates to follow-up recommendations. METHODS: We extracted radiology examination-related data, including report text, for examinations performed between January 1, 2010, and February 28, 2017, from the radiology information system at an academic institution. The data set contained 2,972,164 examinations. The first 6 years were used as the period during which a follow-up recommendation was to be detected, allowing for a maximum of 14 months for a follow-up examination to be performed. RESULTS: At least one recommendation for follow-up imaging was present in 10.6% of radiology reports. Overall, the follow-up imaging adherence rate was 58.14%. Mammography had the highest follow-up adherence rate at 69.03%, followed by MRI at 67.54%. Of the modalities, nuclear medicine had the lowest adherence rate at 37.93%. CONCLUSIONS: This study confirms that follow-up imaging adherence rates are inherently low and vary by modality and that appropriate interventions may be needed to improve compliance to follow-up imaging recommendations.

Detecting Technical Image Quality in Radiology Reports.

Image interpretation accuracy is critical to ensure optimal care, yet many diagnostic reports contain expressions of uncertainty often due to shortcomings in technical quality among other factors. While radiologists will usually attempt to interpret images and render a diagnosis even if the imaging quality is suboptimal, often the details related to any quality concerns are dictated into the report. Despite imaging exam quality being an import factor for accurate image interpretation, there is a significant knowledge gap in terms of understanding the nature and frequency of technical limitations mentioned in radiology reports. To address some of these limitations, in this research we developed algorithms to automatically detect a broad spectrum of acquisition-related quality concerns using a dataset containing 1,210,858 exams. There was some type of a quality concern mentioned in 2.4% of exams with motion being the most frequent.

A model to determine payments associated with radiology procedures.

OBJECTIVE: Across the United States, there is a growing number of patients in Accountable Care Organizations and under risk contracts with commercial insurance. This is due to proliferation of new value-based payment models and care delivery reform efforts. In this context, the business model of radiology within a hospital or health system context is shifting from a primary profit-center to a cost-center with a goal of cost savings. Radiology departments need to increasingly understand how the transactional nature of the business relates to financial rewards. The main challenge with current reporting systems is that the information is presented only at an aggregated level, and often not broken down further, for instance, by type of exam. As such, the primary objective of this research is to provide better visibility into payments associated with individual radiology procedures in order to better calibrate expense/capital structure of the imaging enterprise to the actual revenue or value-add to the organization it belongs to. MATERIALS AND METHODS: We propose a methodology that can be used to determine technical payments at a procedure level. We use a proportion based model to allocate payments to individual radiology procedures based on total charges (which also includes non-radiology related charges). RESULTS: Using a production dataset containing 424,250 radiology exams we calculated the overall average technical charge for Radiology to be $873.08 per procedure and the corresponding average payment to be $326.43 (range: $48.27 for XR and $2750.11 for PET/CT) resulting in an average payment percentage of 37.39% across all exams. DISCUSSION: We describe how charges associated with a procedure can be used to approximate technical payments at a more granular level with a focus on Radiology. The methodology is generalizable to approximate payment for other services as well. Understanding payments associated with each procedure can be useful during strategic practice planning. CONCLUSIONS: Charge-to-total charge ratio can be used to approximate radiology payments at a procedure level.

Predictive modeling to identify scheduled radiology appointments resulting in non-attendance in a hospital setting.

No-show appointments are a troublesome, but frequent, occurrence in radiology hospital departments and private practice. Prior work in medical appointment no-show prediction has focused on general practice and has not considered features specific to the radiology environment. We collect data from 16 years of outpatient examinations in a multi-site hospital radiology department. Data from the radiology information system (RIS) are fused with patient income estimated from U.S. Census data. Features were categorized into three groups: Patient, Exam, and Scheduling. Models based on the total feature set and separately on each feature group were developed using logistic regression to assess the per-appointment likelihood of no-show. After five-fold cross-validation, no-show prediction using the total feature set from 554,611 appointments yielded an area under the curve (AUC) of 0.770 ± 0.003. Feature groups that were most informative in the prediction of no-show appointments were those based on the type of exam and on scheduling attributes such as the lead time of scheduling the appointment. A data-driven no-show prediction model like the one presented here could be useful to schedulers in the implementation of an automated scheduling policy or the assignment of examinations with a high risk of no-show to lower impact appointment slots.

Inpatient Complexity in Radiology-a Practical Application of the Case Mix Index Metric.

With ongoing healthcare payment reforms in the USA, radiology is moving from its current state of a revenue generating department to a new reality of a cost-center. Under bundled payment methods, radiology does not get reimbursed for each and every inpatient procedure, but rather, the hospital gets reimbursed for the entire hospital stay under an applicable diagnosis-related group code. The hospital case mix index (CMI) metric, as defined by the Centers for Medicare and Medicaid Services, has a significant impact on how much hospitals get reimbursed for an inpatient stay. Oftentimes, patients with the highest disease acuity are treated in tertiary care radiology departments. Therefore, the average hospital CMI based on the entire inpatient population may not be adequate to determine department-level resource utilization, such as the number of technologists and nurses, as case length and staffing intensity gets quite high for sicker patients. In this study, we determine CMI for the overall radiology department in a tertiary care setting based on inpatients undergoing radiology procedures. Between April and September 2015, CMI for radiology was 1.93. With an average of 2.81, interventional neuroradiology had the highest CMI out of the ten radiology sections. CMI was consistently higher across seven of the radiology sections than the average hospital CMI of 1.81. Our results suggest that inpatients undergoing radiology procedures were on average more complex in this hospital setting during the time period considered. This finding is relevant for accurate calculation of labor analytics and other predictive resource utilization tools.

Improving Quality of Follow-Up Imaging Recommendations in Radiology.

Failure of timely follow-up imaging recommendations can result in suboptimal patient care. Evidence suggests that the use of conditional language in follow-up recommendations is associated with changes to follow-up compliance. Assuming that referring physicians prefer explicit guidance for follow-up recommendations, we develop algorithms to extract recommended modality and interval from follow-up imaging recommendations related to lung, thyroid and adrenal findings. Using a production dataset of 417,451 radiology reports, we observed that on average, follow-up interval was not mentioned in 79.4% of reports, and modality was missing in 47.4% of reports (4,819 reports contained a follow-up imaging recommendation for one of the three findings). We also developed an interactive dashboard to be used to monitor compliance rates. Recognizing the importance of increasing precision of follow-up recommendations, a quality improvement pilot study is underway with the goal of achieving a target where follow-up modality and interval are both explicitly specified.

Extracting Follow-Up Recommendations and Associated Anatomy from Radiology Reports.

Adherence rates for timely imaging follow-up are usually low due to low rates of diligence by referring physicians and/or patients with following recommendations for follow-up imaging. This can lead to delayed treatment, poor patient outcomes, unnecessary testing, and legal liability. Existing follow-up recommendation detection methods are often disease- and modality-specific. To address some of these limitations, we present a generic radiology report processing pipeline that can be used to extract follow-up imaging recommendations by anatomy using an ontology-based approach. Using a large dataset from three hospitals, we discuss our methodology in the context of identifying follow-up imaging recommendations that are related to lung, adrenal and/or thyroid conditions. The algorithm has 99% accuracy (95% CI: 95.8-99%). We also present an interactive dashboard that can be used to understand trends related to follow-up recommendations.