Proceed with caution: using verbal protocol analysis to measure situation awareness.

Verbal protocol analysis (VPA) is often used to elicit information about the cognitive processes of operators as it provides rich data and can be used in naturalistic settings. Recently VPA has been used to investigate the acquisition and maintenance of situation awareness (SA), and to make comparisons between groups despite a lack of research regarding the efficacy of using VPA for this purpose. This train simulator experiment investigated whether VPA can effectively measure SA. Novice and expert participants were recorded on an audio device while talking aloud throughout the trials and their verbalisations were transcribed verbatim. A coding scheme developed from the transcripts was used to code the verbalisations. Results did not support the use of VPA as a measure of SA but did show that VPA detected differences in SA errors. Potential reasons for the conflicting findings between this experiment and those conducted by other researchers are discussed. Practitioner summary: This paper examined the validity of verbal protocol analysis (VPA) as a situation awareness (SA) measure. A repeated measures experiment was conducted using a train simulator. Normal VPA did not detect changes in SA but a measure of errors did. Caution should be used when using VPA to measure SA. Abbreviations: LETSSA: low-event task subjective situation awareness technique; SA: situation awareness; SAGAT: situation awareness global assessment technique; TPD: train performance display; VPA: verbal protocol analysis.

Normalization of Deviation: Quotation Error in Human Factors.

Objective The objective of this paper is to examine quotation error in human factors. Background Science progresses through building on the work of previous research. This requires accurate quotation. Quotation error has a number of adverse consequences: loss of credibility, loss of confidence in the journal, and a flawed basis for academic debate and scientific progress. Quotation error has been observed in a number of domains, including marine biology and medicine, but there has been little or no previous study of this form of error in human factors, a domain that specializes in the causes and management of error. Methods A study was conducted examining quotation accuracy of 187 extracts from 118 published articles that cited a control article (Vaughan's 1996 book: The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA). Results Of extracts studied, 12.8% ( n = 24) were classed as inaccurate, with 87.2% ( n = 163) being classed as accurate. A second dimension of agreement was examined with 96.3% ( n = 180) agreeing with the control article and only 3.7% ( n = 7) disagreeing. The categories of accuracy and agreement form a two by two matrix. Conclusion Rather than simply blaming individuals for quotation error, systemic factors should also be considered. Vaughan's theory, normalization of deviance, is one systemic theory that can account for quotation error. Application Quotation error is occurring in human factors and should receive more attention. According to Vaughan's theory, the normal everyday systems that promote scholarship may also allow mistakes, mishaps, and quotation error to occur.

Situational pressures on aviation decision making: goal seduction and situation aversion.

INTRODUCTION: The context in which a decision is made can shape the decision process in important ways. Of particular interest here are 'strong' situations, that is, contexts where the situation itself exerts an influence on decision making. Borrowing concepts from field theory and approach/avoidance theory, this paper explores the influence of strong situations on pilot decision making. METHOD: There were 28 pilots flying in Alaska who were interviewed using a critical incident technique where participants were asked to relate stories about challenging flying situations. A bottom-up qualitative analysis of reported social and psychological pressures revealed a range of types, including strong situations, which are described in this paper. RESULTS: Strong situations were mentioned in 67% of the interviews and included situations likely to motivate the pilot toward unsafe behavior (goal seduction) and situations likely to motivate the pilot away from safe behavior (situation aversion). Goal seduction situations included rescuing others, meeting a significant other, time-related constraints, financial pressures, and unspecified goal seduction; aversive situations included physical discomfort, not having maintenance or facilities for the airplane, and a lack of basic necessities (e.g., lodging) for the pilot. DISCUSSION: Goal seduction describes one broad, unspecified influence on pilot decision making, but situation aversion is also an important motivator for some pilots. Goal seduction and situation aversion may well underlie the phenomena of plan continuation errors, the tendency for pilots to continue despite evidence suggesting it is imprudent. This study suggests some specific factors that constrain decision making for pilots in challenging situations.

Organizational pressures and mitigating strategies in small commercial aviation: findings from Alaska.

INTRODUCTION: Recent attention has focused on the way in which organizational factors can erode safety in aviation, particularly in regions that have a high accident rate, such as Alaska. The present study builds on this work by examining the direct and indirect pressures that can be exerted on pilots by Alaskan operators. In addition, the paper examines ways in which organizations and individuals manage the effects of pressure. METHOD: Using the critical incident method to uncover situations where the pilot's skills had been challenged, 28 pilots who flew in Alaska were interviewed. A bottom-up qualitative analysis revealed a range of organizational pressures and mitigating strategies. RESULTS: Pilots in Alaska encountered both implicit and explicit norms and expectations to fly in marginal conditions. Pressure also arose from pilots' awareness of the need for their company to make money and from perceived job competition. Some Alaskan operators were able to mitigate the effects of pressure on their pilots and some pilots reported mitigating pressure to fly by managing their employer's expectations and re-emphasizing safety. DISCUSSION: Organizational factors were found to be an important source of pressure for pilots and are likely to contribute to the high accident rate in Alaska. Balancing the competing demands of safety and productivity may be extremely difficult for many small operators, which places a heavy reliance on the decision making of individuals. Both the subtle pressures on individual pilots and strategies for mitigating those pressures are, therefore, extremely important to safety and productivity in small-scale commercial aviation.

The Low-Event Task Subjective Situation Awareness (LETSSA) technique: Development and evaluation of a new subjective measure of situation awareness.

Situation awareness (SA) is an important component of an individual's ability to function in a complex environment. As such, it is essential to have effective measures of an individual's SA. The most widely used subjective measure of SA is the Situation Awareness Rating Technique [SART]. However, SART has been criticised for not predicting performance or objective SA, and being highly correlated with workload. This paper describes the development and testing of a new subjective measure of SA, the Low-Event Task Subjective Situation Awareness (LETSSA) measure. To evaluate LETSSA a train simulator study was conducted with 23 novice and 26 expert freight train drivers. LETSSA was able to detect differences in manipulated SA and was comparable to an established objective SA measure (SAGAT). LETSSA was significantly associated with performance but not significantly associated with workload. While further validation is required, LETSSA shows promise as an effective subjective measure of SA.

Breakdowns in coordinated decision making at and above the incident management team level: an analysis of three large scale Australian wildfires.

Emergency situations are by their nature difficult to manage and success in such situations is often highly dependent on effective team coordination. Breakdowns in team coordination can lead to significant disruption to an operational response. Breakdowns in coordination were explored in three large-scale bushfires in Australia: the Kilmore East fire, the Wangary fire, and the Canberra Firestorm. Data from these fires were analysed using a top-down and bottom-up qualitative analysis technique. Forty-four breakdowns in coordinated decision making were identified, which yielded 83 disconnects grouped into three main categories: operational, informational and evaluative. Disconnects were specific instances where differences in understanding existed between team members. The reasons why disconnects occurred were largely consistent across the three sets of data. In some cases multiple disconnects occurred in a temporal manner, which suggested some evidence of disconnects creating states that were conducive to the occurrence of further disconnects. In terms of resolution, evaluative disconnects were nearly always resolved however operational and informational disconnects were rarely resolved effectively. The exploratory data analysis and discussion presented here represents the first systematic research to provide information about the reasons why breakdowns occur in emergency management and presents an account of how team processes can act to disrupt coordination and the operational response.