Testing the reliability of accident analysis methods: a comparison of AcciMap, STAMP-CAST and AcciNet.

Accident analysis methods are used to model the multifactorial cause of adverse incidents. Methods such as AcciMap, STAMP-CAST and recently AcciNet, are systemic approaches that support the identification of safety interventions across sociotechnical system levels. Despite their growing popularity, little is known about how reliable systems-based methods are when used to describe, model and classify contributory factors and relationships. Here, we conducted an intra-rater and inter-rater reliability assessment of AcciMap, STAMP-CAST and AcciNet using the Signal Detection Theory (SDT) paradigm. A total of 180 hours' worth of analyses across 360 comparisons were performed by 30 expert analysts. Findings revealed that all three methods produced a weak to moderate positive correlation coefficient, however the inter-rater reliability of STAMP-CAST was significantly higher compared to AcciMap and AcciNet. No statistically significant or practically meaningful differences were found between methods in the overall intra-rater reliability analyses. Implications and future research directions are discussed.

Practitioners who undertake accident analysis within their organisations should consider the use of STAMP-CAST due to the significantly higher inter-rater reliability findings obtained in this study compared to AcciMap and AcciNet, particularly if they tend to work alone and/or part of relatively small teams.

The Chartered Institute of Ergonomics and Human Factors at 75: perspectives on contemporary challenges and future directions for Ergonomics and Human Factors.

As the UK's Chartered Institute of Ergonomics and Human Factors (CIEHF) celebrates its 75th anniversary, it is worth reflecting on our discipline's contribution, current state, and critical future endeavours. We present the perspectives of 18 EHF professionals who were asked to respond to five questions regarding the impact of EHF, contemporary challenges, and future directions. Co-authors were in agreement that EHF's impact has been only limited to date and that critical issues require resolution, such as increasing the number of suitably qualified practitioners, resolving the research-practice gap, and increasing awareness of EHF and its benefits. Frequently discussed future directions include advanced emerging technologies such as artificial intelligence, the development of new EHF methods, and enhancing the quality and reach of education and training. The majority felt there will be a need for EHF in 75 years; however, many noted that our methods will need to adapt to meet new needs.Practitioner statement: This article provides the perspectives of 18 Ergonomics and Human Factors (EHF) professionals on the impact of EHF, contemporary challenges and critical future directions, and changes that are necessary to ensure EHF remains relevant in future. As such, it provides important guidance on future EHF research and practice.

Towards a unified model of accident causation: refining and validating the systems thinking safety tenets.

The systems thinking tenets were developed based on a synthesis of contemporary accident causation theory, models and approaches and encapsulate 15 features of complex systems that interact to create both safety and adverse events. Whilst initial testing provided supportive evidence, the tenets have not yet been subject to formal validation. This article presents the findings from a three-round Delphi study undertaken to refine and validate the tenets and assess their suitability for inclusion in a unified model of accident causation. Participants with expertise in accident causation and systems thinking provided feedback on the tenets and associated definitions until an acceptable level of consensus was achieved. The results reduced the original 15 tenets to 14 and 10 were identified as important to include in unified model of accident causation. The refined systems thinking tenets are presented along with future research directions designed to facilitate their use in safety practice.Practitioner summary: This article presents a refined and validated set of systems thinking tenets which describe features of complex systems that interact to create adverse events. The tenets can be used by practitioners to proactively identify safety leading indicators and contributory factors during adverse event analysis.

Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics.

There is growing interest in the use of systems-based risk assessment methods in Human Factors and Ergonomics (HFE). The purpose of this study was to test the intra-rater reliability and criterion-referenced concurrent validity of three systems-based risk assessment approaches: (i) the Systems-Theoretic Process Analysis (STPA) method; (ii) the Event Analysis of Systemic Teamwork Broken Links (EAST-BL) method; and, (iii) the Network Hazard Analysis and Risk Management System (Net-HARMS) method. Reliability and validity measures were obtained using the Signal Detection Theory (SDT) paradigm. Whilst STPA identified the highest number of risks, the findings indicate a weak to moderate level of reliability and validity for STPA, EAST-BL and Net-HARMS. There were no statistically significant differences between the methods across analyses. The results suggest that there is merit to the continued use of systems-based risk assessment methods following a series of methodological extensions that aim to enhance the reliability and validity of future applications. Practitioner summary The three risk assessment methods produced weak to moderate levels of stability and accuracy regarding their capability to predict risks. There is a pressing need to further test the reliability and validity of safety methods in Human Factors and Ergonomics.

State of science: evolving perspectives on 'human error'.

This paper reviews the key perspectives on human error and analyses the core theories and methods developed and applied over the last 60 years. These theories and methods have sought to improve our understanding of what human error is, and how and why it occurs, to facilitate the prediction of errors and use these insights to support safer work and societal systems. Yet, while this area of Ergonomics and Human Factors (EHF) has been influential and long-standing, the benefits of the 'human error approach' to understanding accidents and optimising system performance have been questioned. This state of science review analyses the construct of human error within EHF. It then discusses the key conceptual difficulties the construct faces in an era of systems EHF. Finally, a way forward is proposed to prompt further discussion within the EHF community. Practitioner statement This state-of-science review discusses the evolution of perspectives on human error as well as trends in the theories and methods applied to understand, prevent and mitigate error. It concludes that, although a useful contribution has been made, we must move beyond a focus on an individual error to systems failure to understand and optimise whole systems.

Complexity theory in accident causation: using AcciMap to identify the systems thinking tenets in 11 catastrophes.

The quest to explain and understand the cause of accidents is both ever-present and ongoing amongst the safety science community. In an attempt to advance the theory and science of accident causation, researchers have recently formalised a set of '15 systems thinking tenets' that cover the conditions and characteristics of work systems that are believed to contribute to the cause of accidents. The purpose of this study was to attempt to identify the systems thinking tenets across a range of different systems and accidents using the Accident Mapping (AcciMap) method. The findings suggest that the tenets can be attributed to play a role in accident causation, however as a result of this process, the capability of AcciMap has been brought into question. Implications and directions for future research are described. Practitioner Summary: This study is an extension of previous work that suggested there was a need to test for the 'systems thinking tenets of accident causation' in a multi-incident dataset. We used AcciMap to evaluate whether it has the capability to support ongoing accident analysis activities in ergonomics research.

Interaction-centred design: an end user evaluation of road intersection concepts developed using the cognitive work analysis design toolkit (CWA-DT).

Crashes at intersections represent an important road safety problem. Interactions between different road user types, such as between vehicles and vulnerable road users, are a particular concern. It has been suggested driver-centric road design plays a role in crashes. A multi-road user evaluation of three novel intersection designs is described. The designs were generated using the Cognitive Work Analysis Design Toolkit, underpinned by sociotechnical systems theory. The desktop evaluation involved drivers, motorcyclists, cyclists and pedestrians rating the design concepts against alignment with design goals, sociotechnical systems theory and usability, and providing feedback on the positive and negative aspects. Two concepts received more positive ratings and feedback in comparison to a concept that provided more user autonomy. The evaluation results also highlight clear differences in needs across road user groups. The design and evaluation process demonstrates how sociotechnical systems values and principles can be applied in the design of public spaces. Practitioner Summary: This study involved a participatory evaluation of novel road intersection designs, based on sociotechnical systems theory. The results identified important differences in needs and preferences across road user groups and demonstrate the value of sociotechnical systems theory and user participation in road transport design and evaluation processes. Abbreviations: CWA-DT: cognitive work analysis design toolkit; WDA: work domain analysis; SUS: system usability scale.

Quantum ergonomics: shifting the paradigm of the systems agenda.

A paradigm is an accepted world view. If we do not continually question our paradigm then wider trends and movements will overtake the discipline leaving it ill adapted to future challenges. This Special Issue is an opportunity to keep systems thinking at the forefront of ergonomics theory and practice. Systems thinking prompts us to ask whether ergonomics, as a discipline, has been too timid? Too preoccupied with the resolution of immediate problems with industrial-age methods when, approaching fast, are developments which could render these operating assumptions an irrelevance. Practical case studies are presented to show how abstract systems problems can be tackled head-on to deliver highly innovative and cost-effective insights. The strategic direction of the discipline foregrounds high-quality systems problems. These are something the discipline is well able to respond to provided that the appropriate operating paradigms are selected. Practitioner Summary: High-quality systems problems are the future of the discipline. How do we convert obtuse sounding systems concepts into practical interventions? In this paper, the essence of systems thinking is distilled and practical case studies used to demonstrate the benefits of this new paradigm.

Fitting methods to paradigms: are ergonomics methods fit for systems thinking?

The issues being tackled within ergonomics problem spaces are shifting. Although existing paradigms appear relevant for modern day systems, it is worth questioning whether our methods are. This paper asks whether the complexities of systems thinking, a currently ubiquitous ergonomics paradigm, are outpacing the capabilities of our methodological toolkit. This is achieved through examining the contemporary ergonomics problem space and the extent to which ergonomics methods can meet the challenges posed. Specifically, five key areas within the ergonomics paradigm of systems thinking are focused on: normal performance as a cause of accidents, accident prediction, system migration, systems concepts and ergonomics in design. The methods available for pursuing each line of inquiry are discussed, along with their ability to respond to key requirements. In doing so, a series of new methodological requirements and capabilities are identified. It is argued that further methodological development is required to provide researchers and practitioners with appropriate tools to explore both contemporary and future problems. Practitioner Summary: Ergonomics methods are the cornerstone of our discipline. This paper examines whether our current methodological toolkit is fit for purpose given the changing nature of ergonomics problems. The findings provide key research and practice requirements for methodological development.

Can Link Analysis Be Applied to Identify Behavioral Patterns in Train Recorder Data?

OBJECTIVE: A proof-of-concept analysis was conducted to establish whether link analysis could be applied to data from on-train recorders to detect patterns of behavior that could act as leading indicators of potential safety issues. BACKGROUND: On-train data recorders capture data about driving behavior on thousands of routine journeys every day and offer a source of untapped data that could be used to offer insights into human behavior. METHOD: Data from 17 journeys undertaken by six drivers on the same route over a 16-hr period were analyzed using link analysis, and four key metrics were examined: number of links, network density, diameter, and sociometric status. RESULTS: The results established that link analysis can be usefully applied to data captured from on-vehicle recorders. The four metrics revealed key differences in normal driver behavior. These differences have promising construct validity as leading indicators. CONCLUSION: Link analysis is one method that could be usefully applied to exploit data routinely gathered by on-vehicle data recorders. It facilitates a proactive approach to safety based on leading indicators, offers a clearer understanding of what constitutes normal driving behavior, and identifies trends at the interface of people and systems, which is currently a key area of strategic risk. APPLICATION: These research findings have direct applications in the field of transport data monitoring. They offer a means of automatically detecting patterns in driver behavior that could act as leading indicators of problems during operation and that could be used in the proactive monitoring of driver competence, risk management, and even infrastructure design.

Using the event analysis of systemic teamwork (EAST) to explore conflicts between different road user groups when making right hand turns at urban intersections.

Collisions between different types of road users at intersections form a substantial component of the road toll. This paper presents an analysis of driver, cyclist, motorcyclist and pedestrian behaviour at intersections that involved the application of an integrated suite of ergonomics methods, the Event Analysis of Systemic Teamwork (EAST) framework, to on-road study data. EAST was used to analyse behaviour at three intersections using data derived from an on-road study of driver, cyclist, motorcyclist and pedestrian behaviour. The analysis shows the differences in behaviour and cognition across the different road user groups and pinpoints instances where this may be creating conflicts between different road users. The role of intersection design in creating these differences in behaviour and resulting conflicts is discussed. It is concluded that currently intersections are not designed in a way that supports behaviour across the four forms of road user studied. Interventions designed to improve intersection safety are discussed. Practitioner Summary: Intersection safety currently represents a key road safety issue worldwide. This paper presents a novel application of a framework of ergonomics methods for studying differences in road user behaviour at intersections. The findings support development of interventions that consider all road users as opposed to one group in isolation.

Exploring schema-driven differences in situation awareness between road users: an on-road study of driver, cyclist and motorcyclist situation awareness.

Collisions between different road users make a substantial contribution to road trauma. Although evidence suggests that different road users interpret the same road situations differently, it is not clear how road users' situation awareness differs, nor is it clear which differences might lead to conflicts. This article presents the findings from an on-road study conducted to examine driver, motorcyclist and cyclist situation awareness in different road environments. The findings suggest that, in addition to minor differences in the structure of different road users' situation awareness (i.e. amount of information and how it is integrated), the actual content of situation awareness in terms of road user schemata, the resulting interaction with the world and the information underpinning situation awareness is markedly different. Further examination indicates that the differences are likely to be compatible along arterial roads, shopping strips and at roundabouts, but that they may create conflicts between different road users at intersections. Interventions designed to support compatible situation awareness and behaviour between different road users are discussed.

It's a small world after all: contrasting hierarchical and edge networks in a simulated intelligence analysis task.

This article presents the rationale behind an important enhancement to a socio-technical model of organisations and teams derived from military research. It combines this with empirical results which take advantage of these enhancements. In Part 1, a new theoretical legacy for the model is developed based on Ergonomics theories and insights. This allows team communications data to be plotted into the model and for it to demonstrate discriminate validity between alternative team structures. Part 2 presents multinational data from the Experimental Laboratory for Investigating Collaboration, Information-sharing, and Trust (ELICIT) community. It was surprising to see that teams in both traditional hierarchical command and control and networked 'peer-to-peer' organisations operate in broadly the same area of the model, a region occupied by networks of communication exhibiting 'small world' properties. Small world networks may be of considerable importance for the Ergonomics analysis of team organisation and performance. PRACTITIONER SUMMARY: This article is themed around macro and systems Ergonomics, and examines the effects of command and control structures. Despite some differences in behaviour and measures of agility, when given the freedom to do so, participants organised themselves into a small world network. This network type has important and interesting implications for the Ergonomics design of teams and organisations.

Festschrift in honour of Professor Neville Stanton: A lone crusader in a world of driving simulators.

The automotive future has always pointed to a world of intelligent co-pilots and robot cars, but perhaps no more so than Knight Rider. In this 1980's television series the fictional Knight Industries Two Thousand (KITT) was a supercomputer on wheels with 1000 megabytes of memory. The protagonist was Michael Knight, a young loner on a crusade to champion the cause of the innocent and the helpless. This was a shadowy flight into the trials and tribulations of different levels of automation, re-claiming control when automation failed, and a wilful, chatty computer co-driver. An amusing metaphor, perhaps, for the research impact made by Neville Stanton in the field of vehicle automation. Without question - to paraphrase the Knight Rider outro - "one man can make a difference". This festschrift in Neville's honour tells the story of how.

What could they have been thinking? How sociotechnical system design influences cognition: a case study of the Stockwell shooting.

Understanding why an individual acted in a certain way is of fundamental importance to the human factors community, especially when the choice of action results in an undesirable outcome. This challenge is typically tackled by applying retrospective interview techniques to generate models of what happened, recording deviations from a 'correct procedure'. While such approaches may have great utility in tightly constrained procedural environments, they are less applicable in complex sociotechnical systems that require individuals to modify procedures in real time to respond to a changing environment. For complex sociotechnical systems, a formative approach is required that maps the information available to the individual and considers its impact on performance and action. A context-specific, activity-independent, constraint-based model forms the basis of this approach. To illustrate, an example of the Stockwell shooting is used, where an innocent man, mistaken for a suicide bomber, was shot dead. Transferable findings are then presented. STATEMENT OF RELEVANCE: This paper presents a new approach that can be applied proactively to consider how sociotechnical system design, and the information available to an individual, can affect their performance. The approach is proposed to be complementary to the existing tools in the mental models phase of the cognitive work analysis framework.

Systems thinking-based risk assessment methods applied to sports performance: A comparison of STPA, EAST-BL, and Net-HARMS in the context of elite women's road cycling.

There is increasing interest in applying systems Human Factors and Ergonomics (HFE) methods in sport. Risk assessment (RA) methods can be used identify risks which may impact the performance of individual athletes, teams, and overall sports systems; however, they have not yet been tested in sport. This study sets out to apply and compare three systems thinking-based RA methods in the context of elite sports performance and report on the frequency and types of the risks identified. The Systems-Theoretic Process Analysis (STPA) method, the Event Analysis of Systemic Teamwork Broken Links (EAST-BL) method, and the Networked Hazard Analysis and Risk Management System (Net-HARMS) method were applied to elite women's road cycling to identify all the credible risks that could degrade optimal team performance. The findings demonstrate that all three methods appear to provide useful results in a context other than safety, and that multiple risks threatening the performance of the cycling team were identified. Whilst the frequency and types of risks differed across the methods applied, there are additional theoretical, methodological, and practical implications to be considered prior to the selection and use of systems thinking-based RA approaches. Recommendations and directions for future HFE and sports science research are discussed.

The famous five factors in teamwork: a case study of fratricide.

The purpose of this paper is to propose foundations for a theory of errors in teamwork based upon analysis of a case study of fratricide alongside a review of the existing literature. This approach may help to promote a better understanding of interactions within complex systems and help in the formulation of hypotheses and predictions concerning errors in teamwork, particularly incidents of fratricide. It is proposed that a fusion of concepts drawn from error models, with common causal categories taken from teamwork models, could allow for an in-depth exploration of incidents of fratricide. It is argued that such a model has the potential to explore the core causal categories identified as present in an incident of fratricide. This view marks fratricide as a process of errors occurring throughout the military system as a whole, particularly due to problems in teamwork within this complex system. Implications of this viewpoint for the development of a new theory of fratricide are offered. STATEMENT OF RELEVANCE: This article provides an insight into the fusion of existing error and teamwork models for the analysis of an incident of fratricide. Within this paper, a number of commonalities among models of teamwork have been identified allowing for the development of a model.

A new approach for designing cognitive artefacts to support disaster management.

The public display of information is not a new phenomenon; whiteboards (and blackboards before them) have commonly been used to share information. Once annotated, these collaborative information sources have clear benefits for cognition, reducing the burden on short-term memory and supporting parallel processing. This paper introduces a structured approach for considering the design and development of these cognitive artefacts with the aim of enhancing system performance. To illustrate this approach, a case study of military support to large-scale civilian emergencies is used. Using the introduced process, a number of displays are developed supporting individual and shared understanding of the domain, enhancing the crisis planning and management process. The case study demonstrates how the proposed structured approach can inform the designer and lead to domain specific designs. A clear trail can be plotted between the analysis of the domain and the development of the cognitive artefacts. STATEMENT OF RELEVANCE: This paper presents a generic approach for the design of cognitive artefacts to enhance system performance. Illustrated by a case study, it is contended that the approach, and adaptations of it, are applicable to supporting the design of information aids for a wide range of complex domains.

Translating concepts of complexity to the field of ergonomics.

Since 1958 more than 80 journal papers from the mainstream ergonomics literature have used either the words 'complex' or 'complexity' in their titles. Of those, more than 90% have been published in only the past 20 years. This observation communicates something interesting about the way in which contemporary ergonomics problems are being understood. The study of complexity itself derives from non-linear mathematics but many of its core concepts have found analogies in numerous non-mathematical domains. Set against this cross-disciplinary background, the current paper aims to provide a similar initial mapping to the field of ergonomics. In it, the ergonomics problem space, complexity metrics and powerful concepts such as emergence raise complexity to the status of an important contingency factor in achieving a match between ergonomics problems and ergonomics methods. The concept of relative predictive efficiency is used to illustrate how this match could be achieved in practice. What is clear overall is that a major source of, and solution to, complexity are the humans in systems. Understanding complexity on its own terms offers the potential to leverage disproportionate effects from ergonomics interventions and to tighten up the often loose usage of the term in the titles of ergonomics papers. STATEMENT OF RELEVANCE: This paper reviews and discusses concepts from the study of complexity and maps them to ergonomics problems and methods. It concludes that humans are a major source of and solution to complexity in systems and that complexity is a powerful contingency factor, which should be considered to ensure that ergonomics approaches match the true nature of ergonomics problems.