Inter-observer agreement and reliability assessment for observational studies of clinical work.

Inter-observer agreement (IOA) is a key aspect of data quality in time-and-motion studies of clinical work. To date, such studies have used simple and ad hoc approaches for IOA assessment, often with minimal reporting of methodological details. The main methodological issues are how to align time-stamped task intervals that rarely have agreeing start and end times, and how to assess IOA for multiple nominal variables. We present a combination of methods that simultaneously addresses both these issues and provides a more appropriate measure by which to assess IOA for time-and-motion studies. The issue of alignment is addressed by converting task-level data into small time windows then aligning data from different observers by time. A method applicable to multivariate nominal data, the iota score, is then applied to the time-aligned data. We illustrate our approach by comparing iota scores to the mean of univariate Cohen's kappa scores through application of these measures to existing data from an observational study of emergency department physicians. While the two scores generated very similar results under certain conditions, iota was more resilient to sparse data issues. Our results suggest that iota applied to time windows considerably improves on previous methods used for IOA assessment in time-and-motion studies, and that Cohen's kappa and other univariate measures should not be considered the gold standard. Rather, there is an urgent need for ongoing explicit discussion of methodological issues and solutions to improve the ways in which data quality is assessed in time-and-motion studies in order to ensure the conclusions drawn from such studies are robust.

Dangers and uses of cross-correlation in analyzing time series in perception, performance, movement, and neuroscience: The importance of constructing transfer function autoregressive models.

Many articles on perception, performance, psychophysiology, and neuroscience seek to relate pairs of time series through assessments of their cross-correlations. Most such series are individually autocorrelated: they do not comprise independent values. Given this situation, an unfounded reliance is often placed on cross-correlation as an indicator of relationships (e.g., referent vs. response, leading vs. following). Such cross-correlations can indicate spurious relationships, because of autocorrelation. Given these dangers, we here simulated how and why such spurious conclusions can arise, to provide an approach to resolving them. We show that when multiple pairs of series are aggregated in several different ways for a cross-correlation analysis, problems remain. Finally, even a genuine cross-correlation function does not answer key motivating questions, such as whether there are likely causal relationships between the series. Thus, we illustrate how to obtain a transfer function describing such relationships, informed by any genuine cross-correlations. We illustrate the confounds and the meaningful transfer functions by two concrete examples, one each in perception and performance, together with key elements of the R software code needed. The approach involves autocorrelation functions, the establishment of stationarity, prewhitening, the determination of cross-correlation functions, the assessment of Granger causality, and autoregressive model development. Autocorrelation also limits the interpretability of other measures of possible relationships between pairs of time series, such as mutual information. We emphasize that further complexity may be required as the appropriate analysis is pursued fully, and that causal intervention experiments will likely also be needed.

Emergency doctors' strategies to manage competing workload demands in an interruptive environment: An observational workflow time study.

An observational workflow time study was conducted involving doctors in the emergency department (ED) of a large Australian hospital. During 121.7 h across 58 sessions, we observed interruptive events, conceptualised as prompts, and doctors' strategies to handle those prompts (task-switching, multitasking, acknowledgement, deferral and deflection) to assess the role of multiple work system factors influencing doctors' work in the ED. Prompt rates varied vastly between work scenarios, being highest during non-verbal solo tasks. The propensity to use certain strategies also differed with task type, prompt type and location within the department, although task-switching was by far the most frequent. Communicative prompts were important in patient treatment and workload management. Clinicians appear to adjust their communication strategies in response to contextual factors in order to deliver patient care. Risk due to the interruptive nature of ED communication is potentially outweighed by the positive effects of timely information transfer and advice provision.

Effects of lowering the legal BAC to 0.08 on single-vehicle-nighttime fatal traffic crashes in 19 jurisdictions.

BACKGROUND: In the past few decades, numerous policies, including those that lower legal blood alcohol concentration limits, have been enacted to reduce alcohol-impaired driving. In the US, 41 states and the District of Columbia have enacted 0.08 per se laws, which specify that if a driver's BAC is at or above 0.08, a violation has occurred even if the driver does not show signs of intoxication. OBJECTIVE: We examined effects of lowering the blood alcohol concentration limit to 0.08 per se on fatal traffic crashes in 18 states and the District of Columbia, and whether effects of the law varied by state or by baseline rates of fatal traffic crashes. METHOD: Data on fatal traffic crashes were obtained from the Fatality Analysis Reporting System, including all states that enacted 0.08 per se prior to 2001 in the contiguous United States. Effects of the 0.08 law were examined in each state separately, and the overall effect across states was examined using a mixed-model Poisson regression on single-vehicle-nighttime fatal traffic crashes. RESULTS: State-specific analyses showed that fatal traffic crashes significantly decreased in three of the 19 states following the introduction of the 0.08 law, prior to adjusting for potential confounders. The mixed-model regression showed a statistically significant 5.2% reduction in single-vehicle-nighttime fatal traffic crashes associated with the 0.08 law across all states, after adjusting for administrative license revocation, the number of Friday and Saturday nights in a month, and trends in all other types of fatal traffic crashes. Findings indicate that the effect of the 0.08 law does not vary significantly by state or baseline rate of fatal traffic crashes in a state, and no significant statistical interaction exists between 0.08 and administrative license revocation policy effects.

Managing competing demands through task-switching and multitasking: a multi-setting observational study of 200 clinicians over 1000 hours.

OBJECTIVE: To provide a detailed characterisation of clinicians' work management strategies. DESIGN: 1002.3 h of observational data were derived from three previous studies conducted in a teaching hospital in Sydney, Australia, among emergency department (ED) doctors (n=40), ward doctors (n=57) and ward nurses (n=104). The rates of task-switching (pausing a task to handle an incoming task) and multitasking (adding a task in parallel to an existing task) were compared in each group. Random intercepts logistic regression was used to determine factors significantly associated with clinicians' use of task-switching over multitasking and to quantify variation between individual clinicians. RESULTS: Task-switching rates were higher among ED doctors (6.0 per hour) than ward staff (2.2 and 1.8 per hour for doctors and nurses, respectively) and vice versa for multitasking rates (9.2 vs 17.3 and 14.1 per hour). Clinicians' strategy use was significantly related to the nature and complexity of work and to the person they were working with. In some settings, time of day, day of the week or previous chosen strategy affected a clinician's strategy. Independent of these factors, there was significant variation between individual clinicians in their use of strategies in a given situation (ED doctors p=0.04, ward staff p=0.03). CONCLUSIONS: Despite differences in factors associated with work management strategy use among ED doctors, ward doctors and ward nurses, clinicians in all settings appeared to prioritise certain types of tasks over others. Documentation was generally given low priority in all groups, while the arrival of direct care tasks tended to be treated with high priority. These findings suggest that considerations of safety may be implicit in task-switching and multitasking decisions. Although these strategies have been cast in a negative light, future research should consider their role in optimising competing quality and efficiency demands.

Changes in nurses' work associated with computerised information systems: Opportunities for international comparative studies using the revised Work Observation Method By Activity Timing (WOMBAT).

An important step in advancing global health through informatics is to understand how systems support health professionals to deliver improved services to patients. Studies in several countries have highlighted the potential for clinical information systems to change patterns of work and communication, and in particular have raised concerns that they reduce nurses' time in direct care. However measuring the effects of systems on work is challenging and comparisons across studies have been hindered by a lack of standardised definitions and measurement tools. This paper describes the Work Observation Method by Activity Time (WOMBAT) technique version 1.0 and the ways in which the data generated can describe different aspects of health professionals' work. In 2011 a revised WOMBAT version 2.0 was developed specifically to facilitate its use by research teams in different countries. The new features provide opportunities for international comparative studies of nurses' work to be conducted.

The impact of interruptions on clinical task completion.

BACKGROUND: Interruptions and multitasking are implicated as a major cause of clinical inefficiency and error. OBJECTIVE: The aim was to measure the association between emergency doctors' rates of interruption and task completion times and rates. METHODS: The authors conducted a prospective observational time and motion study in the emergency department of a 400-bed teaching hospital. Forty doctors (91% of medical staff) were observed for 210.45 h on weekdays. The authors calculated the time on task (TOT); the relationship between TOT and interruptions; and the proportion of time in work task categories. Length-biased sampling was controlled for. RESULTS: Doctors were interrupted 6.6 times/h. 11% of all tasks were interrupted, 3.3% more than once. Doctors multitasked for 12.8% of time. The mean TOT was 1:26 min. Interruptions were associated with a significant increase in TOT. However, when length-biased sampling was accounted for, interrupted tasks were unexpectedly completed in a shorter time than uninterrupted tasks. Doctors failed to return to 18.5% (95% CI 15.9% to 21.1%) of interrupted tasks. CONCLUSIONS: It appears that in busy interrupt-driven clinical environments, clinicians reduce the time they spend on clinical tasks if they experience interruptions, and may delay or fail to return to a significant portion of interrupted tasks. Task shortening may occur because interrupted tasks are truncated to 'catch up' for lost time, which may have significant implications for patient safety.

Association of interruptions with an increased risk and severity of medication administration errors.

BACKGROUND: Interruptions have been implicated as a cause of clinical errors, yet, to our knowledge, no empirical studies of this relationship exist. We tested the hypothesis that interruptions during medication administration increase errors. METHODS: We performed an observational study of nurses preparing and administering medications in 6 wards at 2 major teaching hospitals in Sydney, Australia. Procedural failures and interruptions were recorded during direct observation. Clinical errors were identified by comparing observational data with patients' medication charts. A volunteer sample of 98 nurses (representing a participation rate of 82%) were observed preparing and administering 4271 medications to 720 patients over 505 hours from September 2006 through March 2008. Associations between procedural failures (10 indicators; eg, aseptic technique) and clinical errors (12 indicators; eg, wrong dose) and interruptions, and between interruptions and potential severity of failures and errors, were the main outcome measures. RESULTS: Each interruption was associated with a 12.1% increase in procedural failures and a 12.7% increase in clinical errors. The association between interruptions and clinical errors was independent of hospital and nurse characteristics. Interruptions occurred in 53.1% of administrations (95% confidence interval [CI], 51.6%-54.6%). Of total drug administrations, 74.4% (n = 3177) had at least 1 procedural failure (95% CI, 73.1%-75.7%). Administrations with no interruptions (n = 2005) had a procedural failure rate of 69.6% (n = 1395; 95% CI, 67.6%-71.6%), which increased to 84.6% (n = 148; 95% CI, 79.2%-89.9%) with 3 interruptions. Overall, 25.0% (n = 1067; 95% CI, 23.7%-26.3%) of administrations had at least 1 clinical error. Those with no interruptions had a rate of 25.3% (n = 507; 95% CI, 23.4%-27.2%), whereas those with 3 interruptions had a rate of 38.9% (n = 68; 95% CI, 31.6%-46.1%). Nurse experience provided no protection against making a clinical error and was associated with higher procedural failure rates. Error severity increased with interruption frequency. Without interruption, the estimated risk of a major error was 2.3%; with 4 interruptions this risk doubled to 4.7% (95% CI, 2.9%-7.4%; P < .001). CONCLUSION: Among nurses at 2 hospitals, the occurrence and frequency of interruptions were significantly associated with the incidence of procedural failures and clinical errors.

Optimized parameters for fluorescence-based verification of ballast water exchange by ships.

Mid-ocean ballast water exchange is mandatory for ships discharging foreign ballast in US territorial waters in order to reduce the risk of biological invasions. However, a reliable tool for determining whether the procedure took place is lacking. We investigated chromophoric dissolved organic matter (CDOM) fluorescence as a tracer of mid-ocean exchange on nine research cruises out of Asia, Europe, and the USA, focusing on challenging source conditions (high salinity, low CDOM). Using parallel factor analysis, we identified nine independent fluorescent components present in varying concentrations in the ocean and in ballast water. One component was sufficient for predicting the coastal vs oceanic source of most ballast water samples. Across nine cruises, thresholds (1.7 and 0.7 ppb quinine sulfate equivalent units) at two fixed wavelength pairs (lambda(ex)/lambda(em) = 320/414 and 370/496 nm, respectively) discriminated coastal from oceanic ballast water in > 95% of samples (N = 514). Our results suggest that single- and dual-channel fluorometers could be optimized for verifying ballast water exchange.