Real-time Context-Aware Multimodal Network for Activity and Activity-Stage Recognition from Team Communication in Dynamic Clinical Settings.

In clinical settings, most automatic recognition systems use visual or sensory data to recognize activities. These systems cannot recognize activities that rely on verbal assessment, lack visual cues, or do not use medical devices. We examined speech-based activity and activity-stage recognition in a clinical domain, making the following contributions. (1) We collected a high-quality dataset representing common activities and activity stages during actual trauma resuscitation events-the initial evaluation and treatment of critically injured patients. (2) We introduced a novel multimodal network based on audio signal and a set of keywords that does not require a high-performing automatic speech recognition (ASR) engine. (3) We designed novel contextual modules to capture dynamic dependencies in team conversations about activities and stages during a complex workflow. (4) We introduced a data augmentation method, which simulates team communication by combining selected utterances and their audio clips, and showed that this method contributed to performance improvement in our data-limited scenario. In offline experiments, our proposed context-aware multimodal model achieved F1-scores of 73.2±0.8% and 78.1±1.1% for activity and activity-stage recognition, respectively. In online experiments, the performance declined about 10% for both recognition types when using utterance-level segmentation of the ASR output. The performance declined about 15% when we omitted the utterance-level segmentation. Our experiments showed the feasibility of speech-based activity and activity-stage recognition during dynamic clinical events.

Discovering interpretable medical process models: A case study in trauma resuscitation.

Understanding the actual work (i.e., "work-as-done") rather than theorized work (i.e., "work-as-imagined") during complex medical processes is critical for developing approaches that improve patient outcomes. Although process mining has been used to discover process models from medical activity logs, it often omits critical steps or produces cluttered and unreadable models. In this paper, we introduce a TraceAlignment-based ProcessDiscovery method called TAD Miner to build interpretable process models for complex medical processes. TAD Miner creates simple linear process models using a threshold metric that optimizes the consensus sequence to represent the backbone process, and then identifies both concurrent activities and uncommon-but-critical activities to represent the side branches. TAD Miner also identifies the locations of repeated activities, an essential feature for representing medical treatment steps. We conducted a study using activity logs of 308 pediatric trauma resuscitations to develop and evaluate TAD Miner. TAD Miner was used to discover process models for five resuscitation goals, including establishing intravenous (IV) access, administering non-invasive oxygenation, performing back assessment, administering blood transfusion, and performing intubation. We quantitively evaluated the process models with several complexity and accuracy metrics, and performed qualitative evaluation with four medical experts to assess the accuracy and interpretability of the discovered models. Through these evaluations, we compared the performance of our method to that of two state-of-the-art process discovery algorithms: Inductive Miner and Split Miner. The process models discovered by TAD Miner had lower complexity and better interpretability than the state-of-the-art methods, and the fitness and precision of the models were comparable. We used the TAD process models to identify (1) the errors and (2)the best locations for the tentative steps in knowledge-driven expert models. The knowledge-driven models were revised based on the modifications suggested by the discovered models. The improved modeling using TAD Miner may enhance understanding of complex medical processes.

Factors Affecting Peripheral Intravenous Catheter Placement During Pediatric Trauma Resuscitation.

INTRODUCTION: Intravenous access is required for resuscitation of injured patients but may be delayed in children because of challenges associated with peripheral intravenous (PIV) catheter placement. Early identification of factors predisposing patients to difficult PIV placement can assist in deciding strategies for timely intravenous access. METHODS: We conducted a retrospective, video-based review of injured children and adolescents treated between April 2018 and May 2019. Patient demographic, physiological, injury, and resuscitation characteristics were obtained from the patient record, including age, race, weight, injury type, Injury Severity Score, initial systolic blood pressure, initial Glasgow Coma Score, intubation status, activation level, and presence of prearrival notification. Video review was used to determine the time to PIV placement, the number of attempts required, the purpose for additional access, and the reason for abandonment of PIV placement. Multivariable regressions were used to determine factors associated with successful placement. RESULTS: During the study period, 154 consented patients underwent attempts at PIV placement in the trauma bay. Placement was successful in 139 (90.3%) patients. Older patients (OR [odds ratio]: 0.9, 95% confidence interval [CI]: 0.9, 0.9) and patients who required the highest level activation response (OR: 0.0, 95% CI: 0.0, 0.3) were less likely to have an attempt at PIV placement abandoned. Children with nonblunt injuries (OR: 11.6, 95% CI: 1.3, 119.2) and pre-existing access (OR: 39.6, 95% CI: 7.0, 350.6) were more likely to have an attempt at PIV placement abandoned. Among patients with successful PIV placement, the time required for establishing PIV access was faster as age increased (-0.5 s, 95% CI: -1.1, -0.0). CONCLUSIONS: Younger age was associated with abandonment of PIV attempts and, when successful, increased time to placement. Strategies to improve successful PIV placement and alternate routes of access should be considered early to prevent treatment delays in younger children.

Analysis of Task Attributes Associated with Crisis Checklist Compliance in Pediatric Trauma Resuscitation.

Although checklists can improve overall team performance during medical crises, non-compliant checklist use poses risks to patient safety. We examined how task attributes affected checklist compliance by studying the use of a digital checklist during trauma resuscitation. We first determined task attributes and checklist compliance behaviors for 3,131 resuscitation tasks. Using statistical analyses and qualitative video review, we then identified barriers to accurately tracking task status, finding that certain task attributes were associated with non-compliant checklist behaviors. For example, tasks with multiple steps were more likely to be incorrectly recorded as completed when the task was not performed to completion. We discuss challenges in capturing and tracking the status of tasks with attributes that contribute to non-compliant checklist use. We also contribute a framework for understanding how tasks with certain attributes can be designed on checklists to improve compliance.

An Analysis of Speech during Life Saving Interventions to Inform the Design of a Computerized System for Delay Detection.

We describe an analysis of speech during time-critical, team-based medical work and its potential to indicate process delays. We analyzed speech intention and sentence types during 39 trauma resuscitations with delays in one of three major lifesaving interventions: intravenous/intraosseous (IV/IO) line insertion, cardiopulmonary and resuscitation (CPR), and intubation. We found a significant difference in patterns of speech during delays vs. speech during non-delayed work. The speech intention during CPR delays, however, differed from the other LSIs, suggesting that context of speech must be considered. These findings will inform the design of a clinical decision support system (CDSS) that will use multiple sensor modalities to alert medical teams to delays in real time. We conclude with design implications and challenges associated with speech-based activity recognition in complex medical processes.

Real-time medical phase recognition using long-term video understanding and progress gate method.

We introduce a real-time system for recognizing five phases of the trauma resuscitation process, the initial management of injured patients in the emergency department. We used depth videos as input to preserve the privacy of the patients and providers. The depth videos were recorded using a Kinect-v2 mounted on the sidewall of the room. Our dataset consisted of 183 depth videos of trauma resuscitations. The model was trained on 150 cases with more than 30 minutes each and tested on the remaining 33 cases. We introduced a reduced long-term operation (RLO) method for extracting features from long segments of video and combined it with the regular model having short-term information only. The model with RLO outperformed the regular short-term model by 5% using the accuracy score. We also introduced a progress gate (PG) method to distinguish visually similar phases using video progress. The final system achieved 91% accuracy and significantly outperformed previous systems for phase recognition in this setting.

Personal Protective Equipment Adherence of Pediatric Resuscitation Team Members During the COVID-19 Pandemic.

STUDY OBJECTIVE: During the COVID-19 pandemic, health care workers have had the highest risk of infection among essential workers. Although personal protective equipment (PPE) use is associated with lower infection rates, appropriate use of PPE has been variable among health care workers, even in settings with COVID-19 patients. We aimed to evaluate the patterns of PPE adherence during emergency department resuscitations that included aerosol-generating procedures. METHODS: We conducted a retrospective, video-based review of pediatric resuscitations involving one or more aerosol-generating procedures during the first 3 months of the COVID-19 pandemic in the United States (March to June 2020). Recommended adherence (complete, inadequate, absent) with 5 PPE items (headwear, eyewear, masks, gowns, gloves) and the duration of potential exposure were evaluated for individuals in the room after aerosol-generating procedure initiation. RESULTS: Among the 345 health care workers observed during 19 resuscitations, 306 (88.7%) were nonadherent (inadequate or absent adherence) with the recommended use of at least 1 PPE type at some time during the resuscitation, 23 (6.7%) of whom had no PPE. One hundred and forty health care workers (40.6%) altered or removed at least 1 type of PPE during the event. The aggregate time in the resuscitation room for health care workers across all events was 118.7 hours. During this time, providers had either absent or inadequate eyewear for 46.4 hours (39.1%) and absent or inadequate masks for 35.2 hours (29.7%). CONCLUSION: Full adherence with recommended PPE use was limited in a setting at increased risk for SARS-CoV-2 virus aerosolization. In addition to ensuring appropriate donning, approaches are needed for ensuring ongoing adherence with PPE recommendations during exposure.

Metrics of shock in pediatric trauma patients: A systematic search and review.

INTRODUCTION: Shock-index (SI) and systolic blood pressure (SBP) are metrics for identifying children and adults with hemodynamic instability following injury. The purpose of this systematic review was to assess the quality of these metrics as predictors of outcomes following pediatric injury. MATERIALS AND METHODS: We conducted a literature search in Pubmed, SCOPUS, and CINAHL to identify studies describing the association between shock metrics on the morbidity and mortality of injured children and adolescents. We used the data presented in the studies to calculate the sensitivity and specificity for each metric. This study was registered with Prospero, protocol CRD42020162971. RESULTS: Fifteen articles met the inclusion criteria. seven studies evaluated SI or SIPA score, an age-corrected version of SI, as predictors of outcomes following pediatric trauma, with one study comparing SIPA score and SBP and one study comparing SI and SBP. The remaining eight studies evaluated SBP as the primary indicator of shock. The median sensitivity for predicting mortality and need for blood transfusion was highest for SI, followed by SIPA, and then SBP. The median specificity for predicting these outcomes was highest for SBP, followed by SIPA, and then SI. CONCLUSIONS: Common conclusions were that high SIPA scores were more specific than SI and more sensitive than SBP. SIPA score had better discrimination for severely injured children compared to SI and SBP. An elevated SIPA was associated with a greater need for blood transfusion and higher in-hospital mortality. SIPA is specific enough to exclude most patients who do not require a blood transfusion.

Association of Ventilation during Initial Trauma Resuscitation for Traumatic Brain Injury and Post-Traumatic Outcomes: A Systematic Review.

OBJECTIVES: In the absence of evidence of acute cerebral herniation, normal ventilation is recommended for patients with traumatic brain injury (TBI). Despite this recommendation, ventilation strategies vary during the initial management of patients with TBI and may impact outcome. The goal of this systematic review was to define the best evidence-based practice of ventilation management during the initial resuscitation period. METHODS: A literature search of PubMed, CINAHL, and SCOPUS identified studies from 2009 through 2019 addressing the effects of ventilation during the initial post-trauma resuscitation on patient outcomes. RESULTS: The initial search yielded 899 articles, from which 13 were relevant and selected for full-text review. Six of the 13 articles met the inclusion criteria, all of which reported on patients with TBI. Either end-tidal carbon dioxide (ETCO2) or partial pressure carbon dioxide (PCO2) were the independent variables associated with mortality. Decreased rates of mortality were reported in patients with normal PCO2 or ETCO2. CONCLUSIONS: Normoventilation, as measured by ETCO2 or PCO2, is associated with decreased mortality in patients with TBI. Preventing hyperventilation or hypoventilation in patients with TBI during the early resuscitation phase could improve outcome after TBI.

Assessment of Non-Routine Events and Significant Physiological Disturbances during Emergency Department Evaluation after Pediatric Head Trauma.

Outcomes following pediatric traumatic brain injury (TBI) are dependent on initial injury severity and prevention of secondary injury. Hypoxia, hypotension, and hyperventilation following TBI are associated with increased mortality. The purpose of this study was to determine the association of non-routine events (NREs) during the initial resuscitation phase with these physiological disturbances. We conducted a video review of pediatric trauma resuscitations of patients with suspected TBI and Glasgow Coma Scale (GCS) scores <13. NREs were rated as "momentary" if task progression was delayed by <1 min and "moderate" if delayed by >1 min. Vital sign monitor data were used to identify periods of significant physiological disturbances. We calculated the association between the rate of overall and moderate NREs per case and the proportion of cases with abnormal vital signs using multi-variate linear regression, controlling for GCS score and need for intubation. Among 26 resuscitations, 604 NREs were identified with a median of 23 (interquartile range [IQR] 17-27.8, range 5-44) per case. Moderate delay NREs occurred in 19 resuscitations (n = 32, median 1 NRE/resuscitation, IQR 0.3-1, range 0-5). Oxygen desaturation and respiratory depression were associated with a greater rate of moderate NREs (p = 0.008, p < 0.001, respectively). We observed no association between duration of hypotension, desaturation, and respiratory depression and overall NRE rate. NREs are common in the initial resuscitation of children with moderate to severe TBI. Episodes of hypoxia and respiratory depression are associated with NREs that cause a moderate delay in task progression. Conformance with resuscitation guidelines is needed to prevent physiological events associated with adverse outcomes following pediatric TBI.

VidTr: Video Transformer Without Convolutions.

We introduce Video Transformer (VidTr) with separable-attention for video classification. Comparing with commonly used 3D networks, VidTr is able to aggregate spatio-temporal information via stacked attentions and provide better performance with higher efficiency. We first introduce the vanilla video transformer and show that transformer module is able to perform spatio-temporal modeling from raw pixels, but with heavy memory usage. We then present VidTr which reduces the memory cost by 3.3× while keeping the same performance. To further optimize the model, we propose the standard deviation based topK pooling for attention (pooltopK_std), which reduces the computation by dropping non-informative features along temporal dimension. VidTr achieves state-of-the-art performance on five commonly used datasets with lower computational requirement, showing both the efficiency and effectiveness of our design. Finally, error analysis and visualization show that VidTr is especially good at predicting actions that require long-term temporal reasoning.

Multi-Label Activity Recognition using Activity-specific Features and Activity Correlations.

Multi-label activity recognition is designed for recognizing multiple activities that are performed simultaneously or sequentially in each video. Most recent activity recognition networks focus on single-activities, that assume only one activity in each video. These networks extract shared features for all the activities, which are not designed for multi-label activities. We introduce an approach to multi-label activity recognition that extracts independent feature descriptors for each activity and learns activity correlations. This structure can be trained end-to-end and plugged into any existing network structures for video classification. Our method outperformed state-of-the-art approaches on four multi-label activity recognition datasets. To better understand the activity-specific features that the system generated, we visualized these activity-specific features in the Charades dataset. The code will be released later.

Designing Interactive Alerts to Improve Recognition of Critical Events in Medical Emergencies.

Vital sign values during medical emergencies can help clinicians recognize and treat patients with life-threatening injuries. Identifying abnormal vital signs, however, is frequently delayed and the values may not be documented at all. In this mixed-methods study, we designed and evaluated a two-phased visual alert approach for a digital checklist in trauma resuscitation that informs users about undocumented vital signs. Using an interrupted time series analysis, we compared documentation in the periods before (two years) and after (four months) the introduction of the alerts. We found that introducing alerts led to an increase in documentation throughout the post-intervention period, with clinicians documenting vital signs earlier. Interviews with users and video review of cases showed that alerts were ineffective when clinicians engaged less with the checklist or set the checklist down to perform another activity. From these findings, we discuss approaches to designing alerts for dynamic team-based settings.

Classification strategies for non-routine events occurring in high-risk patient care settings: A scoping review.

INTRODUCTION: Non-routine events (NREs) are atypical or unusual occurrences in a pre-defined process. Although some NREs in high-risk clinical settings have no adverse effects on patient care, others can potentially cause serious patient harm. A unified strategy for identifying and describing NREs in these domains will facilitate the comparison of results between studies. METHODS: We conducted a literature search in PubMed, CINAHL, and EMBASE to identify studies related to NREs in high-risk domains and evaluated the methods used for event observation and description. We applied The Joint Commission on Accreditation of Healthcare Organization (JCAHO) taxonomy (cause, impact, domain, type, prevention, and mitigation) to the descriptions of NREs from the literature. RESULTS: We selected 25 articles that met inclusion criteria for review. Real-time documentation of NREs was more common than a retrospective video review. Thirteen studies used domain experts as observers and seven studies validated observations with interrater reliability. Using the JCAHO taxonomy, "cause" was the most frequently applied classification method, followed by "impact," "type," "domain," and "prevention and mitigation." CONCLUSIONS: NREs are frequent in high-risk medical settings. Strengths identified in several studies included the use of multiple observers with domain expertise and validation of the event ascertainment approach using interrater reliability. By applying the JCAHO taxonomy to the current literature, we provide an example of a structured approach that can be used for future analyses of NREs.

Assessment of Nonroutine Events During Intubation After Pediatric Trauma.

BACKGROUND: Intubation in the early postinjury phase can be a high-risk procedure associated with an increased risk of mortality when delayed. Nonroutine events (NREs) are workflow disruptions that can be latent safety threats in high-risk settings and may contribute to adverse outcomes. MATERIALS AND METHODS: We reviewed videos of intubations of injured children (age<17 y old) in the emergency department occurring between 2014 and 2018 to identify NREs occurring between the decision to intubate and successful intubation ("critical window"). RESULTS: Among 34 children requiring intubation, the indications included GCS≤8 (n = 20, 58.8%), cardiac arrest (n = 6, 17.6%), airway protection (n = 5, 14.7%), and respiratory failure (n = 3, 8.8%). The median duration of the "critical window" was 7.5 min (range 1.4-27.5 min), with a median of six NREs per case in this period (range 2-30). Most NREs (n = 159, 61.9%) delayed workflow, with 31 (12.1%) of these delays each lasting more than one minute. Eighty-seven NREs (33.9%) had a potential for harm but did not lead to direct patient harm. The most common NREs directly related to the intubation process were poor positioning for intubation (n = 23, 8.9%) and difficulty passing the endotracheal tube (n = 5, 1.9%), with most being attributed to the anesthesiologist performing the intubation (n = 51, range 0-7). CONCLUSIONS: Workflow disruptions related to nonroutine events were frequent during pediatric trauma intubation and were often associated with delays and potential for patient harm. Interventions for improving the efficiency and timeliness of the critical window should focus on adherence to intubation protocol and improving communication and teamwork related to tasks in this phase.

RESIDUAL RECURRENT NEURAL NETWORK FOR SPEECH ENHANCEMENT.

Most current speech enhancement models use spectrogram features that require an expensive transformation and result in phase information loss. Previous work has overcome these issues by using convolutional networks to learn the temporal correlations across high-resolution waveforms. These models, however, are limited by memory-intensive dilated convolution and aliasing artifacts from upsampling. We introduce an end-to-end fully recurrent neural network for single-channel speech enhancement. The network structured as an hourglass-shape that can efficiently capture long-range temporal dependencies by reducing the features resolution without information loss. Also, we use residual connections to prevent gradient decay over layers and improve the model generalization. Experimental results show that our model outperforms state-of-the-art approaches in six quantitative evaluation metrics.

Speech-Based Activity Recognition for Trauma Resuscitation.

We present a speech-based approach to recognize team activities in the context of trauma resuscitation. We first analyzed the audio recordings of trauma resuscitations in terms of activity frequency, noise-level, and activity-related keyword frequency to determine the dataset characteristics. We next evaluated different audio-preprocessing parameters (spectral feature types and audio channels) to find the optimal configuration. We then introduced a novel neural network to recognize the trauma activities using a modified VGG network that extracts features from the audio input. The output of the modified VGG network is combined with the output of a network that takes keyword text as input, and the combination is used to generate activity labels. We compared our system with several baselines and performed a detailed analysis of the performance results for specific activities. Our results show that our proposed architecture that uses Mel-spectrum spectral coefficients features with a stereo channel and activity-specific frequent keywords achieve the highest accuracy and average F1-score.

Video-based Concurrent Activity Recognition for Trauma Resuscitation.

We introduce a video-based system for concurrent activity recognition during teamwork in a clinical setting. During system development, we preserved patient and provider privacy by pre-computing spatio-temporal features. We extended the inflated 3D ConvNet (i3D) model for concurrent activity recognition. For the model training, we tuned the weights of the final stages of i3D using back-propagated loss from the fully-connected layer. We applied filtering on the model predictions to remove noisy predictions. We evaluated the system on five activities performed during trauma resuscitation, the initial management of injured patients in the emergency department. Our system achieved an average value of 74% average precision (AP) for these five activities and outperformed previous systems designed for the same domain. We visualized feature maps from the model, showing that the system learned to focus on regions relevant to performance of each activity.

Process conformance is associated with successful first intubation attempt and lower odds of adverse events in a paediatric emergency setting.

BACKGROUND: Intubation is an essential, life-saving skill but associated with a high risk for adverse outcomes. Intubation protocols have been implemented to increase success and reduce complications, but the impact of protocol conformance is not known. Our study aimed to determine association between conformance with an intubation process model and outcomes. METHODS: An interdisciplinary expert panel developed a process model of tasks and sequencing deemed necessary for successful intubation. The model was then retrospectively used to review videos of intubations from 1 February, 2014, to 31 January, 2016, in a paediatric emergency department at a time when no process model or protocol was in existence. RESULTS: We evaluated 113 patients, 77 (68%) were successfully intubated on first attempt. Model conformance was associated with a higher likelihood of first attempt success when using direct laryngoscopy (OR 1.09, 95% CI 1.01 to 1.18). The use of video laryngoscopy was associated with an overall higher likelihood of success on first attempt (OR 2.54, 95% CI 1.10 to 5.88). Thirty-seven patients (33%) experienced adverse events. Model conformance was the only factor associated with a lower odds of adverse events (OR 0.94, 95% CI 0.88 to 0.99). CONCLUSIONS: Conformance with a task-based expert-derived process model for emergency intubation was associated with a higher rate of success on first intubation attempt when using direct laryngoscopy and a lower odds of associated adverse events. Further evaluation of the impact of human factors, such as teamwork and decision-making, on intubation process conformance and success and outcomes is needed.

Association Between Prearrival Notification Time and Advanced Trauma Life Support Protocol Adherence.

BACKGROUND: Prearrival notification of injured patients facilitates preparation of personnel, equipment, and other resources needed for trauma evaluation and treatment. Our purpose was to determine the impact of prearrival notification time on adherence to Advanced Trauma Life Support (ATLS) protocols. MATERIALS AND METHODS: Pediatric trauma activations of admitted patients were analyzed by video review to determine activities performed before and after patient arrival. Using an expert model based on ATLS, fitness scores were calculated that represented model adherence, ranging from "0" (noncompliant) to "100" (completely compliant). Multivariate regression was used to determine the association between fitness values of the evaluation phases and the length of prearrival notification time and injury profiles. RESULTS: Ninety-four patients met study criteria. The average overall fitness was 89.0 ± 7.3, with similar fitness values being observed for the primary and secondary surveys (91.5 ± 13.4 and 88.6 ± 7.7, respectively). Prearrival notification time ranged from 67.3 min before to 4.8 min after patient arrival. Longer prearrival notification time was associated with improved completion of prearrival tasks, overall resuscitation performance, and secondary survey performance. The positive association of overall and secondary survey fitness with notification time was no longer observed when notification time was <5 min and <10 min, respectively. Notification time was correlated with a higher percentage of required team members when the patient arrived (Pearson correlation coefficient 0.46, P < 0.001). CONCLUSIONS: Prearrival notification time has a significant impact on adherence to ATLS protocol. Strategies for improving notification time or improving performance when adequate notification cannot be achieved are needed.