JSSE: Joint Sequential Semantic Encoder for Zero-Shot Event Recognition.

Zero-shot learning (ZSL) is a paradigm in transfer learning that aims to recognize unknown categories by having a mere description of them. The problem of ZSL has been thoroughly studied in the domain of static object recognition, however, ZSL for dynamic events (ZSER) such as activities and gestures has hardly been investigated. In this context, this paper addresses ZSER by relying on semantic attributes of events to transfer the learned knowledge from seen classes to unseen ones. First, we utilized the Amazon Mechanical Turk platform to create the first attribute-based gesture dataset, referred to as ZSGL, comprising the categories present in MSRC and Italian gesture datasets. Overall, our ZSGL dataset consisted of 26 categories, 65 discriminative attributes, and 16 attribute annotations and 400 examples per category. We used trainable recurrent networks and 3D CNNs to learn the spatio-temporal features. Next, we propose a simple yet effective end-to-end approach for ZSER, referred to as Joint Sequential Semantic Encoder (JSSE), to explore temporal patterns, to efficiently represent events in the latent space, and to simultaneously optimize for both the semantic and classification tasks. We evaluate our model on ZSGL and two action datasets (UCF and HMDB), and compared the performance of JSSE against several existing baselines in four experimental conditions: 1. Within-category, 2. Across-category, 3. Closed-set, and 4. Open-Set. Results show that JSSE considerably outperforms (p<0.05) other approaches and performs favorably for both the datasets in all experimental conditions.

Agreement Study Using Gesture Description Analysis.

Choosing adequate gestures for touchless interfaces is a challenging task that has a direct impact on human-computer interaction. Such gestures are commonly determined by the designer, ad-hoc, rule-based or agreement-based methods. Previous approaches to assess agreement grouped the gestures into equivalence classes and ignored the integral properties that are shared between them. In this work, we propose a generalized framework that inherently incorporates the gesture descriptors into the agreement analysis (GDA). In contrast to previous approaches, we represent gestures using binary description vectors and allow them to be partially similar. In this context, we introduce a new metric referred to as Soft Agreement Rate ( S A R ) to measure the level of agreement and provide a mathematical justification for this metric. Further, we performed computational experiments to study the behavior of S A R and demonstrate that existing agreement metrics are a special case of our approach. Our method was evaluated and tested through a guessability study conducted with a group of neurosurgeons. Nevertheless, our formulation can be applied to any other user-elicitation study. Results show that the level of agreement obtained by S A R is 2.64 times higher than the previous metrics. Finally, we show that our approach complements the existing agreement techniques by generating an artificial lexicon based on the most agreed properties.

Preference elicitation: Obtaining gestural guidelines for PACS in neurosurgery.

OBJECTIVE: Accessing medical records is an integral part of neurosurgical procedures in the Operating Room (OR). Gestural interfaces can help reduce the risks for infections by allowing the surgical staff to browse Picture Archiving and Communication Systems (PACS) without touch. The main objectives of this work are to: a) Elicit gestures from neurosurgeons to analyze their preferences, b) Develop heuristics for gestural interfaces, and c) Produce a lexicon that maximizes surgeons' preferences. MATERIALS AND METHODS: A gesture elicitation study was conducted with nine neurosurgeons. Initially, subjects were asked to outline the gestures on a drawing board for each of the PACS commands. Next, the subjects performed one of three imaging tasks using gestures instead of the keyboard and mouse. Each gesture was annotated with respect to the presence/absence of gesture descriptors. Next, K-nearest neighbor approach was used to obtain the final lexicon that complies with the preferred/popular descriptors. RESULTS: The elicitation study resulted in nine gesture lexicons, each comprised of 28 gestures. A paired t-test between the popularity of the overall gesture and the top three descriptors showed that the latter is significantly higher than the former (89.5%-59.7% vs 19.4%, p < 0.001), meaning more than half of the subjects agreed on these descriptors. Next, the gesture heuristics were generated for each command using the popular descriptors. Lastly, we developed a lexicon that complies with surgeons' preferences. CONCLUSIONS: Neurosurgeons do agree on fundamental characteristics of gestures to perform image manipulation tasks. The proposed heuristics could potentially guide the development of future gesture-based interaction of PACS for the OR.

Gestures for Picture Archiving and Communication Systems (PACS) operation in the operating room: Is there any standard?

OBJECTIVE: Gestural interfaces allow accessing and manipulating Electronic Medical Records (EMR) in hospitals while keeping a complete sterile environment. Particularly, in the Operating Room (OR), these interfaces enable surgeons to browse Picture Archiving and Communication System (PACS) without the need of delegating functions to the surgical staff. Existing gesture based medical interfaces rely on a suboptimal and an arbitrary small set of gestures that are mapped to a few commands available in PACS software. The objective of this work is to discuss a method to determine the most suitable set of gestures based on surgeon's acceptability. To achieve this goal, the paper introduces two key innovations: (a) a novel methodology to incorporate gestures' semantic properties into the agreement analysis, and (b) a new agreement metric to determine the most suitable gesture set for a PACS. MATERIALS AND METHODS: Three neurosurgical diagnostic tasks were conducted by nine neurosurgeons. The set of commands and gesture lexicons were determined using a Wizard of Oz paradigm. The gestures were decomposed into a set of 55 semantic properties based on the motion trajectory, orientation and pose of the surgeons' hands and their ground truth values were manually annotated. Finally, a new agreement metric was developed, using the known Jaccard similarity to measure consensus between users over a gesture set. RESULTS: A set of 34 PACS commands were found to be a sufficient number of actions for PACS manipulation. In addition, it was found that there is a level of agreement of 0.29 among the surgeons over the gestures found. Two statistical tests including paired t-test and Mann Whitney Wilcoxon test were conducted between the proposed metric and the traditional agreement metric. It was found that the agreement values computed using the former metric are significantly higher (p < 0.001) for both tests. CONCLUSIONS: This study reveals that the level of agreement among surgeons over the best gestures for PACS operation is higher than the previously reported metric (0.29 vs 0.13). This observation is based on the fact that the agreement focuses on main features of the gestures rather than the gestures themselves. The level of agreement is not very high, yet indicates a majority preference, and is better than using gestures based on authoritarian or arbitrary approaches. The methods described in this paper provide a guiding framework for the design of future gesture based PACS systems for the OR.