Evaluating White Matter Lesion Segmentations with Refined Sørensen-Dice Analysis.

The Sørensen-Dice index (SDI) is a widely used measure for evaluating medical image segmentation algorithms. It offers a standardized measure of segmentation accuracy which has proven useful. However, it offers diminishing insight when the number of objects is unknown, such as in white matter lesion segmentation of multiple sclerosis (MS) patients. We present a refinement for finer grained parsing of SDI results in situations where the number of objects is unknown. We explore these ideas with two case studies showing what can be learned from our two presented studies. Our first study explores an inter-rater comparison, showing that smaller lesions cannot be reliably identified. In our second case study, we demonstrate fusing multiple MS lesion segmentation algorithms based on the insights into the algorithms provided by our analysis to generate a segmentation that exhibits improved performance. This work demonstrates the wealth of information that can be learned from refined analysis of medical image segmentations.

Longitudinal multiple sclerosis lesion segmentation: Resource and challenge.

In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of five subjects with a mean of 4.4 time-points, and test data of fourteen subjects with a mean of 4.4 time-points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state-of-the-art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website2 as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters.

Feasibility of monitoring compliance to the My 5 Moments and Entry/Exit hand hygiene methods in US hospitals.

We compared the ability to observe hand hygiene opportunities using the World Health Organization My 5 Moments method to the Entry/Exit method. Under covert direct observation, Entry/Exit method opportunities were observed at all times. My 5 Moments were observable in 32.3% of episodes, with a lower rate in wards versus intensive care units (28.0% vs 39.4%; P < .01). In US hospitals, the Entry/Exit method appears to be more feasible for directly observed hand hygiene compliance monitoring due to line-of-sight issues and other barriers.

Establishing evidence-based criteria for directly observed hand hygiene compliance monitoring programs: a prospective, multicenter cohort study.

OBJECTIVE: Hand hygiene surveillance programs that rely on direct observations of healthcare worker activity may be limited by the Hawthorne effect. In addition, comparing compliance rates from period to period requires adequately sized samples of observations. We aimed to statistically determine whether the Hawthorne effect is stable over an observation period and statistically derive sample sizes of observations necessary to compare compliance rates. DESIGN: Prospective multicenter cohort study. SETTING: Five intensive care units and 6 medical/surgical wards in 3 geographically distinct acute care hospitals. METHODS: Trained observers monitored hand hygiene compliance during routine care in fixed 1-hour periods, using a standardized collection tool. We estimated the impact of the Hawthorne effect using empirical fluctuation processes and F tests for structural change. Standard sample-size calculation methods were used to estimate how many hand hygiene opportunities are required to accurately measure hand hygiene across various levels of baseline and target compliance. RESULTS: Exit hand hygiene compliance increased after 14 minutes of observation (from 56.2% to 60.5%; P < .001) and increased further after 50 minutes (from 60.5% to 66.0%; P < .001). Entry compliance increased after 38 minutes (from 40.4% to 43.4%; P = .005). Between 79 and 723 opportunities are required during each period, depending on baseline compliance rates (range, 35%-90%) and targeted improvement (5% or 10%). CONCLUSIONS: Limiting direct observation periods to approximately 15 minutes to minimize the Hawthorne effect and determining required number of hand hygiene opportunities observed per period on the basis of statistical power calculations would be expected to improve the validity of hand hygiene surveillance programs.