Testing the reliability of accident analysis methods: a comparison of AcciMap, STAMP-CAST and AcciNet.

Accident analysis methods are used to model the multifactorial cause of adverse incidents. Methods such as AcciMap, STAMP-CAST and recently AcciNet, are systemic approaches that support the identification of safety interventions across sociotechnical system levels. Despite their growing popularity, little is known about how reliable systems-based methods are when used to describe, model and classify contributory factors and relationships. Here, we conducted an intra-rater and inter-rater reliability assessment of AcciMap, STAMP-CAST and AcciNet using the Signal Detection Theory (SDT) paradigm. A total of 180 hours' worth of analyses across 360 comparisons were performed by 30 expert analysts. Findings revealed that all three methods produced a weak to moderate positive correlation coefficient, however the inter-rater reliability of STAMP-CAST was significantly higher compared to AcciMap and AcciNet. No statistically significant or practically meaningful differences were found between methods in the overall intra-rater reliability analyses. Implications and future research directions are discussed.

Practitioners who undertake accident analysis within their organisations should consider the use of STAMP-CAST due to the significantly higher inter-rater reliability findings obtained in this study compared to AcciMap and AcciNet, particularly if they tend to work alone and/or part of relatively small teams.

The Chartered Institute of Ergonomics and Human Factors at 75: perspectives on contemporary challenges and future directions for Ergonomics and Human Factors.

As the UK's Chartered Institute of Ergonomics and Human Factors (CIEHF) celebrates its 75th anniversary, it is worth reflecting on our discipline's contribution, current state, and critical future endeavours. We present the perspectives of 18 EHF professionals who were asked to respond to five questions regarding the impact of EHF, contemporary challenges, and future directions. Co-authors were in agreement that EHF's impact has been only limited to date and that critical issues require resolution, such as increasing the number of suitably qualified practitioners, resolving the research-practice gap, and increasing awareness of EHF and its benefits. Frequently discussed future directions include advanced emerging technologies such as artificial intelligence, the development of new EHF methods, and enhancing the quality and reach of education and training. The majority felt there will be a need for EHF in 75 years; however, many noted that our methods will need to adapt to meet new needs.Practitioner statement: This article provides the perspectives of 18 Ergonomics and Human Factors (EHF) professionals on the impact of EHF, contemporary challenges and critical future directions, and changes that are necessary to ensure EHF remains relevant in future. As such, it provides important guidance on future EHF research and practice.

Analysing dynamic work systems using DynEAST: a demonstration of concept.

The capability of current Ergonomics methods to capture dynamism is limited, stifling our understanding of work-as-done, distributed situational awareness and organisational drift. This paper provides a demonstration of concept of DynEAST; an extension of the EAST framework underpinned by principles from Dynamic Network Analysis, to capture elements of dynamism within work systems. The DynEAST concept is applied to a railway maintenance case study. Case study findings demonstrate how DynEAST outputs can be used to advance our understanding of the aforementioned phenomena and better equip practitioners for current and future Ergonomics challenges.Practitioner summary: This paper introduces the DynEAST method. DynEAST enables HF/E practitioners to model and analyse dynamic features of complex work systems. The development of DynEAST is timely due to the concurrent proliferation of increasingly complex sociotechnical systems and stagnation of HF/E methods development; particularly those able to model systemic dynamism. Abbreviations: DynEAST: dynamic event analysis of systemic teamwork; EAST: dynamic event analysis of systemic teamwork; HF/E: human factors and ergonomics; HF: human factors; DNA: dynamic network analysis; HTA: hierarchal task analysis; CWA: cognitive work analysis; CAST: causal analysis based on system theory; STAMP: system theoretic accident model and processes; FRAM: functional resonance analysis method; SNA: social network analysis; DSA: distributed situational awareness; PPO: possession protection officer; PO: protection officer; RTS: railway track signals; LPA: local possession authority; SMEs: subject matter experts.

Towards a unified model of accident causation: refining and validating the systems thinking safety tenets.

The systems thinking tenets were developed based on a synthesis of contemporary accident causation theory, models and approaches and encapsulate 15 features of complex systems that interact to create both safety and adverse events. Whilst initial testing provided supportive evidence, the tenets have not yet been subject to formal validation. This article presents the findings from a three-round Delphi study undertaken to refine and validate the tenets and assess their suitability for inclusion in a unified model of accident causation. Participants with expertise in accident causation and systems thinking provided feedback on the tenets and associated definitions until an acceptable level of consensus was achieved. The results reduced the original 15 tenets to 14 and 10 were identified as important to include in unified model of accident causation. The refined systems thinking tenets are presented along with future research directions designed to facilitate their use in safety practice.Practitioner summary: This article presents a refined and validated set of systems thinking tenets which describe features of complex systems that interact to create adverse events. The tenets can be used by practitioners to proactively identify safety leading indicators and contributory factors during adverse event analysis.

Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics.

There is growing interest in the use of systems-based risk assessment methods in Human Factors and Ergonomics (HFE). The purpose of this study was to test the intra-rater reliability and criterion-referenced concurrent validity of three systems-based risk assessment approaches: (i) the Systems-Theoretic Process Analysis (STPA) method; (ii) the Event Analysis of Systemic Teamwork Broken Links (EAST-BL) method; and, (iii) the Network Hazard Analysis and Risk Management System (Net-HARMS) method. Reliability and validity measures were obtained using the Signal Detection Theory (SDT) paradigm. Whilst STPA identified the highest number of risks, the findings indicate a weak to moderate level of reliability and validity for STPA, EAST-BL and Net-HARMS. There were no statistically significant differences between the methods across analyses. The results suggest that there is merit to the continued use of systems-based risk assessment methods following a series of methodological extensions that aim to enhance the reliability and validity of future applications. Practitioner summary The three risk assessment methods produced weak to moderate levels of stability and accuracy regarding their capability to predict risks. There is a pressing need to further test the reliability and validity of safety methods in Human Factors and Ergonomics.

The Binary-Based Model (BBM) for Improved Human Factors Method Selection.

OBJECTIVE: This paper presents the Binary-Based Model (BBM), a new approach to Human Factors (HF) method selection. The BBM helps practitioners select the most appropriate HF methodology in relation to the complexity within the target system. BACKGROUND: There are over 200 HF methods available to the practitioner and little guidance to help choose between them. METHOD: The BBM defines a HF "problem space" comprising three complexity attributes. HF problems can be rated against these attributes and located in the "problem space." In addition, a similar HF "approach space" in which 66 predictive methods are rated according to their ability to confront those attributes is defined. These spaces are combined into a "utility space" in which problems and methods coexist. In the utility space, the match between HF problems and methods can be formally assessed. RESULTS: The method space is split into octants to establish broad groupings of methods distributed throughout the space. About 77% of the methods reside in Octant 1 which corresponds to problems with low levels of complexity. This demonstrates that most HF methods are suited to problems in low-complexity systems. CONCLUSION: The location of 77% of the rated methods in Octant 1 indicates that HF practitioners are underserved with methods for analysis of HF problems exhibiting high complexity. APPLICATION: The BBM can be used by multidisciplinary teams to select the most appropriate HF methodology for the problem under analysis. All the materials and analysis are placed in the public domain for modification and consensus building by the wider HF community.

State of science: evolving perspectives on 'human error'.

This paper reviews the key perspectives on human error and analyses the core theories and methods developed and applied over the last 60 years. These theories and methods have sought to improve our understanding of what human error is, and how and why it occurs, to facilitate the prediction of errors and use these insights to support safer work and societal systems. Yet, while this area of Ergonomics and Human Factors (EHF) has been influential and long-standing, the benefits of the 'human error approach' to understanding accidents and optimising system performance have been questioned. This state of science review analyses the construct of human error within EHF. It then discusses the key conceptual difficulties the construct faces in an era of systems EHF. Finally, a way forward is proposed to prompt further discussion within the EHF community. Practitioner statement This state-of-science review discusses the evolution of perspectives on human error as well as trends in the theories and methods applied to understand, prevent and mitigate error. It concludes that, although a useful contribution has been made, we must move beyond a focus on an individual error to systems failure to understand and optimise whole systems.

Complexity theory in accident causation: using AcciMap to identify the systems thinking tenets in 11 catastrophes.

The quest to explain and understand the cause of accidents is both ever-present and ongoing amongst the safety science community. In an attempt to advance the theory and science of accident causation, researchers have recently formalised a set of '15 systems thinking tenets' that cover the conditions and characteristics of work systems that are believed to contribute to the cause of accidents. The purpose of this study was to attempt to identify the systems thinking tenets across a range of different systems and accidents using the Accident Mapping (AcciMap) method. The findings suggest that the tenets can be attributed to play a role in accident causation, however as a result of this process, the capability of AcciMap has been brought into question. Implications and directions for future research are described. Practitioner Summary: This study is an extension of previous work that suggested there was a need to test for the 'systems thinking tenets of accident causation' in a multi-incident dataset. We used AcciMap to evaluate whether it has the capability to support ongoing accident analysis activities in ergonomics research.

Radical systems thinking and the future role of computational modelling in ergonomics: an exploration of agent-based modelling.

We are teetering on the precipice of the imminent Fourth Industrial Revolution. In this new age, systems are set to become more densely intraconnected and interconnected, and massive sociotechnical systems exhibiting unprecedented levels of complexity will increasingly take hold. At the dawning of this new age, the Ergonomics discipline must reflect on its preparedness for tackling problems in these novel systems. This paper engages in this reflection by putting forth a critical commentary on the implication of these changes on the discipline and discusses the utility of our current methods in this new paradigm. A resulting Radical Systems Thinking in Ergonomics Manifesto is put forward - a set of mandates to guide practitioners and researchers in the development of new methods capable of coping with these imminent challenges. From the manifesto are derived a series of capability requirements for future computational modelling approaches in Ergonomics. Practitioner summary: The goal of this paper was to inspire the Ergonomics community to pursue further applications involving computational modelling approaches such as Agent-Based Modelling. It presents a manifesto for the future of the discipline, and from this the capabilities that future computational modelling approaches need to possess. Abbreviations: 1IR: first industrial revolution; 2IR: second industrial revolution; 3IR: third industrial revolution; 4IR: fourth industrial revolution; ABM: agent based model; AI: artificial intelligence; AoF: allocation of function; CPA: cyber physical attack; CPS: cyber-physical system; CWA: cognitive work analysis; DDoS: distributed denial of service; EAST: event analysis of systemic teamwork; FRAM; functional resonance analysis method; HCI: human-computer interaction; HERA: human error and recovery assessment; HET: human error template; HMC: human-machine cooperation; IoT: internet of things; RSTEM: radical systems thinking in ergonomics manifesto; SAI: situated artificial intelligence; STAMP: systems theoretic accident model and processes; TRACEr: technique for the retrospective and predictive analysis of cognitive errors in air traffic control.

Interaction-centred design: an end user evaluation of road intersection concepts developed using the cognitive work analysis design toolkit (CWA-DT).

Crashes at intersections represent an important road safety problem. Interactions between different road user types, such as between vehicles and vulnerable road users, are a particular concern. It has been suggested driver-centric road design plays a role in crashes. A multi-road user evaluation of three novel intersection designs is described. The designs were generated using the Cognitive Work Analysis Design Toolkit, underpinned by sociotechnical systems theory. The desktop evaluation involved drivers, motorcyclists, cyclists and pedestrians rating the design concepts against alignment with design goals, sociotechnical systems theory and usability, and providing feedback on the positive and negative aspects. Two concepts received more positive ratings and feedback in comparison to a concept that provided more user autonomy. The evaluation results also highlight clear differences in needs across road user groups. The design and evaluation process demonstrates how sociotechnical systems values and principles can be applied in the design of public spaces. Practitioner Summary: This study involved a participatory evaluation of novel road intersection designs, based on sociotechnical systems theory. The results identified important differences in needs and preferences across road user groups and demonstrate the value of sociotechnical systems theory and user participation in road transport design and evaluation processes. Abbreviations: CWA-DT: cognitive work analysis design toolkit; WDA: work domain analysis; SUS: system usability scale.

Quantum ergonomics: shifting the paradigm of the systems agenda.

A paradigm is an accepted world view. If we do not continually question our paradigm then wider trends and movements will overtake the discipline leaving it ill adapted to future challenges. This Special Issue is an opportunity to keep systems thinking at the forefront of ergonomics theory and practice. Systems thinking prompts us to ask whether ergonomics, as a discipline, has been too timid? Too preoccupied with the resolution of immediate problems with industrial-age methods when, approaching fast, are developments which could render these operating assumptions an irrelevance. Practical case studies are presented to show how abstract systems problems can be tackled head-on to deliver highly innovative and cost-effective insights. The strategic direction of the discipline foregrounds high-quality systems problems. These are something the discipline is well able to respond to provided that the appropriate operating paradigms are selected. Practitioner Summary: High-quality systems problems are the future of the discipline. How do we convert obtuse sounding systems concepts into practical interventions? In this paper, the essence of systems thinking is distilled and practical case studies used to demonstrate the benefits of this new paradigm.

Fitting methods to paradigms: are ergonomics methods fit for systems thinking?

The issues being tackled within ergonomics problem spaces are shifting. Although existing paradigms appear relevant for modern day systems, it is worth questioning whether our methods are. This paper asks whether the complexities of systems thinking, a currently ubiquitous ergonomics paradigm, are outpacing the capabilities of our methodological toolkit. This is achieved through examining the contemporary ergonomics problem space and the extent to which ergonomics methods can meet the challenges posed. Specifically, five key areas within the ergonomics paradigm of systems thinking are focused on: normal performance as a cause of accidents, accident prediction, system migration, systems concepts and ergonomics in design. The methods available for pursuing each line of inquiry are discussed, along with their ability to respond to key requirements. In doing so, a series of new methodological requirements and capabilities are identified. It is argued that further methodological development is required to provide researchers and practitioners with appropriate tools to explore both contemporary and future problems. Practitioner Summary: Ergonomics methods are the cornerstone of our discipline. This paper examines whether our current methodological toolkit is fit for purpose given the changing nature of ergonomics problems. The findings provide key research and practice requirements for methodological development.

Can Link Analysis Be Applied to Identify Behavioral Patterns in Train Recorder Data?

OBJECTIVE: A proof-of-concept analysis was conducted to establish whether link analysis could be applied to data from on-train recorders to detect patterns of behavior that could act as leading indicators of potential safety issues. BACKGROUND: On-train data recorders capture data about driving behavior on thousands of routine journeys every day and offer a source of untapped data that could be used to offer insights into human behavior. METHOD: Data from 17 journeys undertaken by six drivers on the same route over a 16-hr period were analyzed using link analysis, and four key metrics were examined: number of links, network density, diameter, and sociometric status. RESULTS: The results established that link analysis can be usefully applied to data captured from on-vehicle recorders. The four metrics revealed key differences in normal driver behavior. These differences have promising construct validity as leading indicators. CONCLUSION: Link analysis is one method that could be usefully applied to exploit data routinely gathered by on-vehicle data recorders. It facilitates a proactive approach to safety based on leading indicators, offers a clearer understanding of what constitutes normal driving behavior, and identifies trends at the interface of people and systems, which is currently a key area of strategic risk. APPLICATION: These research findings have direct applications in the field of transport data monitoring. They offer a means of automatically detecting patterns in driver behavior that could act as leading indicators of problems during operation and that could be used in the proactive monitoring of driver competence, risk management, and even infrastructure design.

A novel approach to evaluating the UK childhood immunisation schedule: estimating the effective coverage vector across the entire vaccine programme.

BACKGROUND: The availability of new vaccines can prompt policy makers to consider changes to the routine childhood immunisation programme in the UK. Alterations to one aspect of the schedule may have implications for other areas of the programme (e.g. adding more injections could reduce uptake of vaccines featuring later in the schedule). Colleagues at the Department of Health (DH) in the UK therefore wanted to know whether assessing the impact across the entire programme of a proposed change to the UK schedule could lead to different decisions than those made on the current case-by-case basis. This work is a first step towards addressing this question. METHODS: A novel framework for estimating the effective coverage against all of the diseases within a vaccination programme was developed. The framework was applied to the current (August 2015) UK childhood immunisation programme, plausible extensions to it in the foreseeable future (introducing vaccination against Meningitis B and/or Hepatitis B) and a "what-if" scenario regarding a Hepatitis B vaccine scare that was developed in close collaboration with DH. RESULTS: Our applications of the framework demonstrate that a programme-view of hypothetical changes to the schedule is important. For example, we show how introducing Hepatitis B vaccination could negatively impact aspects of the current programme by reducing uptake of vaccines featuring later in the schedule, and illustrate that the potential benefits of introducing any new vaccine are susceptible to behaviour changes affecting uptake (e.g. a vaccine scare). We show how it may be useful to consider the potential benefits and scheduling needs of all vaccinations on the horizon of interest rather than those of an individual vaccine in isolation, e.g. how introducing Meningitis B vaccination could saturate the early (2-month) visit, thereby potentially restricting scheduling options for Hepatitis B immunisation should it be introduced to the programme in the future. CONCLUSIONS: Our results demonstrate the potential benefit of considering the programme-wide impact of changes to an immunisation schedule, and our framework is an important step in the development of a means for systematically doing so.

Exploring schema-driven differences in situation awareness between road users: an on-road study of driver, cyclist and motorcyclist situation awareness.

Collisions between different road users make a substantial contribution to road trauma. Although evidence suggests that different road users interpret the same road situations differently, it is not clear how road users' situation awareness differs, nor is it clear which differences might lead to conflicts. This article presents the findings from an on-road study conducted to examine driver, motorcyclist and cyclist situation awareness in different road environments. The findings suggest that, in addition to minor differences in the structure of different road users' situation awareness (i.e. amount of information and how it is integrated), the actual content of situation awareness in terms of road user schemata, the resulting interaction with the world and the information underpinning situation awareness is markedly different. Further examination indicates that the differences are likely to be compatible along arterial roads, shopping strips and at roundabouts, but that they may create conflicts between different road users at intersections. Interventions designed to support compatible situation awareness and behaviour between different road users are discussed.

Using the event analysis of systemic teamwork (EAST) to explore conflicts between different road user groups when making right hand turns at urban intersections.

Collisions between different types of road users at intersections form a substantial component of the road toll. This paper presents an analysis of driver, cyclist, motorcyclist and pedestrian behaviour at intersections that involved the application of an integrated suite of ergonomics methods, the Event Analysis of Systemic Teamwork (EAST) framework, to on-road study data. EAST was used to analyse behaviour at three intersections using data derived from an on-road study of driver, cyclist, motorcyclist and pedestrian behaviour. The analysis shows the differences in behaviour and cognition across the different road user groups and pinpoints instances where this may be creating conflicts between different road users. The role of intersection design in creating these differences in behaviour and resulting conflicts is discussed. It is concluded that currently intersections are not designed in a way that supports behaviour across the four forms of road user studied. Interventions designed to improve intersection safety are discussed. Practitioner Summary: Intersection safety currently represents a key road safety issue worldwide. This paper presents a novel application of a framework of ergonomics methods for studying differences in road user behaviour at intersections. The findings support development of interventions that consider all road users as opposed to one group in isolation.

It's a small world after all: contrasting hierarchical and edge networks in a simulated intelligence analysis task.

This article presents the rationale behind an important enhancement to a socio-technical model of organisations and teams derived from military research. It combines this with empirical results which take advantage of these enhancements. In Part 1, a new theoretical legacy for the model is developed based on Ergonomics theories and insights. This allows team communications data to be plotted into the model and for it to demonstrate discriminate validity between alternative team structures. Part 2 presents multinational data from the Experimental Laboratory for Investigating Collaboration, Information-sharing, and Trust (ELICIT) community. It was surprising to see that teams in both traditional hierarchical command and control and networked 'peer-to-peer' organisations operate in broadly the same area of the model, a region occupied by networks of communication exhibiting 'small world' properties. Small world networks may be of considerable importance for the Ergonomics analysis of team organisation and performance. PRACTITIONER SUMMARY: This article is themed around macro and systems Ergonomics, and examines the effects of command and control structures. Despite some differences in behaviour and measures of agility, when given the freedom to do so, participants organised themselves into a small world network. This network type has important and interesting implications for the Ergonomics design of teams and organisations.

Festschrift in honour of Professor Neville Stanton: A lone crusader in a world of driving simulators.

The automotive future has always pointed to a world of intelligent co-pilots and robot cars, but perhaps no more so than Knight Rider. In this 1980's television series the fictional Knight Industries Two Thousand (KITT) was a supercomputer on wheels with 1000 megabytes of memory. The protagonist was Michael Knight, a young loner on a crusade to champion the cause of the innocent and the helpless. This was a shadowy flight into the trials and tribulations of different levels of automation, re-claiming control when automation failed, and a wilful, chatty computer co-driver. An amusing metaphor, perhaps, for the research impact made by Neville Stanton in the field of vehicle automation. Without question - to paraphrase the Knight Rider outro - "one man can make a difference". This festschrift in Neville's honour tells the story of how.

What could they have been thinking? How sociotechnical system design influences cognition: a case study of the Stockwell shooting.

Understanding why an individual acted in a certain way is of fundamental importance to the human factors community, especially when the choice of action results in an undesirable outcome. This challenge is typically tackled by applying retrospective interview techniques to generate models of what happened, recording deviations from a 'correct procedure'. While such approaches may have great utility in tightly constrained procedural environments, they are less applicable in complex sociotechnical systems that require individuals to modify procedures in real time to respond to a changing environment. For complex sociotechnical systems, a formative approach is required that maps the information available to the individual and considers its impact on performance and action. A context-specific, activity-independent, constraint-based model forms the basis of this approach. To illustrate, an example of the Stockwell shooting is used, where an innocent man, mistaken for a suicide bomber, was shot dead. Transferable findings are then presented. STATEMENT OF RELEVANCE: This paper presents a new approach that can be applied proactively to consider how sociotechnical system design, and the information available to an individual, can affect their performance. The approach is proposed to be complementary to the existing tools in the mental models phase of the cognitive work analysis framework.