Towards Single Camera Human 3D-Kinematics.

Markerless estimation of 3D Kinematics has the great potential to clinically diagnose and monitor movement disorders without referrals to expensive motion capture labs; however, current approaches are limited by performing multiple de-coupled steps to estimate the kinematics of a person from videos. Most current techniques work in a multi-step approach by first detecting the pose of the body and then fitting a musculoskeletal model to the data for accurate kinematic estimation. Errors in training data of the pose detection algorithms, model scaling, as well the requirement of multiple cameras limit the use of these techniques in a clinical setting. Our goal is to pave the way toward fast, easily applicable and accurate 3D kinematic estimation. To this end, we propose a novel approach for direct 3D human kinematic estimation D3KE from videos using deep neural networks. Our experiments demonstrate that the proposed end-to-end training is robust and outperforms 2D and 3D markerless motion capture based kinematic estimation pipelines in terms of joint angles error by a large margin (35% from 5.44 to 3.54 degrees). We show that D3KE is superior to the multi-step approach and can run at video framerate speeds. This technology shows the potential for clinical analysis from mobile devices in the future.

Locoregional recurrence rate and disease-specific survival following marginal vs segmental resection for oral squamous cell carcinoma with mandibular bone invasion.

BACKGROUND AND OBJECTIVES: To determine locoregional recurrence rate (LRR) and disease-specific survival (DSS) following marginal vs segmental mandibulectomy. METHODS: Included were 210 patients, who had marginal or segmental mandibulectomy between 2000 and 2017. Marginal resection was performed when complete removal of the tumor was deemed feasible on the condition that at least 1 cm bone height of the inferior border of the mandible could be preserved. Segmental resection was performed in case less than 1 cm bone height of the mandible would remain. Clinical and histopathological data were collected from medical records. LRR and DSS were computed using Kaplan-Meier analysis. Cox-regression analysis was used to identify risk factors for LRR and DSS. RESULTS: A total of 59 marginal and 151 segmental resections had been performed. There was no significant difference in 3- and 5-year LRR (P = .904) and no significant difference in 3- and 5-year DSS (P = .362) between the marginal and segmental resection group. Cox-regression analysis showed a trend for surgical margin less than equal to 1 mm, to affect LRR (P = .05) and surgical margin less than equal 1 mm, perineural invasion and lymph node metastasis to affect DSS (P < .05). CONCLUSIONS: There was no difference in outcome between the two types of mandibulectomy.

Is the lingual fracture line influenced by the mandibular canal or the mylohyoid groove during a bilateral sagittal split osteotomy? A human cadaveric study.

PURPOSE: Although the bilateral sagittal split osteotomy (BSSO) is a routinely performed procedure, exact control of the lingual fracture line remains problematic. The purpose of this study was to determine the various lingual splitting patterns in cadaveric human mandibles after a BSSO and the possible influence of the mandibular canal and mylohyoid groove on the lingual fracture line. MATERIALS AND METHODS: The investigators designed and implemented a case series to compare different lingual fracture lines. A standardized SSO was performed on 40 cadaveric hemimandibles using elevators and splitting forceps. The primary outcome variable during this study was the lingual fracture pattern possibly influenced by independent variables: the mandibular canal, the mylohyoid groove, and dental status. Descriptive and analytic statistics were computed for each study variable. RESULTS: Most lingual fractures (72.5%) ended in the mandibular foramen. Only 25% of fractures were "true" Hunsuck fractures, and no "bad splits" occurred. In addition, 35% of lingual fractures ran more than halfway or entirely through the mandibular canal, whereas only 30% of fractures ran along the mylohyoid groove. However, when the lingual fracture ran along this groove, it had a 6-fold greater chance of ending in the mandibular foramen. CONCLUSIONS: The hypothesis that the mandibular canal or mylohyoid groove would function as the path of least resistance was only partly confirmed. The use of splitters and separators did not increase the incidence of bad splits compared with the literature.

Assessing facial weakness in myasthenia gravis with facial recognition software and deep learning.

OBJECTIVE: Myasthenia gravis (MG) is an autoimmune disease leading to fatigable muscle weakness. Extra-ocular and bulbar muscles are most commonly affected. We aimed to investigate whether facial weakness can be quantified automatically and used for diagnosis and disease monitoring. METHODS: In this cross-sectional study, we analyzed video recordings of 70 MG patients and 69 healthy controls (HC) with two different methods. Facial weakness was first quantified with facial expression recognition software. Subsequently, a deep learning (DL) computer model was trained for the classification of diagnosis and disease severity using multiple cross-validations on videos of 50 patients and 50 controls. Results were validated using unseen videos of 20 MG patients and 19 HC. RESULTS: Expression of anger (p = 0.026), fear (p = 0.003), and happiness (p < 0.001) was significantly decreased in MG compared to HC. Specific patterns of decreased facial movement were detectable in each emotion. Results of the DL model for diagnosis were as follows: area under the curve (AUC) of the receiver operator curve 0.75 (95% CI 0.65-0.85), sensitivity 0.76, specificity 0.76, and accuracy 76%. For disease severity: AUC 0.75 (95% CI 0.60-0.90), sensitivity 0.93, specificity 0.63, and accuracy 80%. Results of validation, diagnosis: AUC 0.82 (95% CI: 0.67-0.97), sensitivity 1.0, specificity 0.74, and accuracy 87%. For disease severity: AUC 0.88 (95% CI: 0.67-1.0), sensitivity 1.0, specificity 0.86, and accuracy 94%. INTERPRETATION: Patterns of facial weakness can be detected with facial recognition software. Second, this study delivers a 'proof of concept' for a DL model that can distinguish MG from HC and classifies disease severity.

Assessment of Parkinson's Disease Severity From Videos Using Deep Architectures.

Parkinson's disease (PD) diagnosis is based on clinical criteria, i.e., bradykinesia, rest tremor, rigidity, etc. Assessment of the severity of PD symptoms with clinical rating scales, however, is subject to inter-rater variability. In this paper, we propose a deep learning based automatic PD diagnosis method using videos to assist the diagnosis in clinical practices. We deploy a 3D Convolutional Neural Network (CNN) as the baseline approach for the PD severity classification and show the effectiveness. Due to the lack of data in clinical field, we explore the possibility of transfer learning from non-medical dataset and show that PD severity classification can benefit from it. To bridge the domain discrepancy between medical and non-medical datasets, we let the network focus more on the subtle temporal visual cues, i.e., the frequency of tremors, by designing a Temporal Self-Attention (TSA) mechanism. Seven tasks from the Movement Disorders Society - Unified PD rating scale (MDS-UPDRS) part III are investigated, which reveal the symptoms of bradykinesia and postural tremors. Furthermore, we propose a multi-domain learning method to predict the patient-level PD severity through task-assembling. We show the effectiveness of TSA and task-assembling method on our PD video dataset empirically. We achieve the best MCC of 0.55 on binary task-level and 0.39 on three-class patient-level classification.

Resolution Learning in Deep Convolutional Networks Using Scale-Space Theory.

Resolution in deep convolutional neural networks (CNNs) is typically bounded by the receptive field size through filter sizes, and subsampling layers or strided convolutions on feature maps. The optimal resolution may vary significantly depending on the dataset. Modern CNNs hard-code their resolution hyper-parameters in the network architecture which makes tuning such hyper-parameters cumbersome. We propose to do away with hard-coded resolution hyper-parameters and aim to learn the appropriate resolution from data. We use scale-space theory to obtain a self-similar parametrization of filters and make use of the N-Jet: a truncated Taylor series to approximate a filter by a learned combination of Gaussian derivative filters. The parameter σ of the Gaussian basis controls both the amount of detail the filter encodes and the spatial extent of the filter. Since σ is a continuous parameter, we can optimize it with respect to the loss. The proposed N-Jet layer achieves comparable performance when used in state-of-the art architectures, while learning the correct resolution in each layer automatically. We evaluate our N-Jet layer on both classification and segmentation, and we show that learning σ is especially beneficial when dealing with inputs at multiple sizes.

Downregulation of SERPINB13 expression in head and neck squamous cell carcinomas associates with poor clinical outcome.

Tumorigenesis of head and neck squamous cell carcinomas (HNSCC) is associated with various genetic changes such as loss of heterozygosity (LOH) on human chromosome 18q21. This chromosomal region maps a gene cluster coding for a family of intracellular serine protease inhibitors (serpins), including SERPINB13. As SERPINB13 expression in HNSCC has recently been shown to be downregulated both at the mRNA and protein levels, here we investigated if such a low SERPINB13 expression is associated with histopathological and clinical parameters of HNSCC tumors and patient survival. By generating specific antibodies followed by immunohistochemistry on a well-defined cohort of 99 HNSCC of the oral cavity and oropharynx, SERPINB13 expression was found to be partially or totally downregulated in 75% of the HNSCC as compared with endogenous expression in non-neoplastic epithelial cells. Downregulation of SERPINB13 protein expression in HNSCC was significantly associated with the presence of LOH at the SERPINB13 gene in the tumors (p = 0.006), a poor differentiation grade of the tumors (p = 0.001), the presence of a lymph node metastasis (p = 0.012), and a decreased disease-free (p = 0.033) as well as overall (p = 0.018) survival of the patients. This is the first report demonstrating that downregulation of SERPINB13 protein expression in HNSCC is positively associated with poor clinical outcome. Therefore, SERPINB13 seems to act as an important protease inhibitor involved in the progression of HNSCC.

Divide and Count: Generic Object Counting by Image Divisions.

We propose a general object counting method that does not use any prior category information. We learn from local image divisions to predict global image-level counts without using any form of local annotations. Our method separates the input image into a sets of image divisions - each fully covering the image. Each image division is composed of a set of region proposals or uniform grid cells. Our approach learns in an endto- end deep learning architecture to predict global image-level counts from local image divisions. The method incorporates a counting layer which predicts object counts in the complete image, by enforcing consistency in counts when dealing with overlapping image regions. Our counting layer is based on the inclusion-exclusion principle from set theory. We analyze the individual building blocks of our proposed approach on Pascal- VOC2007 and evaluate our method on the MS-COCO large scale generic object dataset as well as on three class-specific counting datasets: UCSD pedestrian dataset, and CARPK and PUCPR+ car datasets.

DeepEyes: Progressive Visual Analytics for Designing Deep Neural Networks.

Deep neural networks are now rivaling human accuracy in several pattern recognition problems. Compared to traditional classifiers, where features are handcrafted, neural networks learn increasingly complex features directly from the data. Instead of handcrafting the features, it is now the network architecture that is manually engineered. The network architecture parameters such as the number of layers or the number of filters per layer and their interconnections are essential for good performance. Even though basic design guidelines exist, designing a neural network is an iterative trial-and-error process that takes days or even weeks to perform due to the large datasets used for training. In this paper, we present DeepEyes, a Progressive Visual Analytics system that supports the design of neural networks during training. We present novel visualizations, supporting the identification of layers that learned a stable set of patterns and, therefore, are of interest for a detailed analysis. The system facilitates the identification of problems, such as superfluous filters or layers, and information that is not being captured by the network. We demonstrate the effectiveness of our system through multiple use cases, showing how a trained network can be compressed, reshaped and adapted to different problems.

Con-Text: Text Detection for Fine-Grained Object Classification.

This paper focuses on fine-grained object classification using recognized scene text in natural images. While the state-of-the-art relies on visual cues only, this paper is the first work which proposes to combine textual and visual cues. Another novelty is the textual cue extraction. Unlike the state-of-the-art text detection methods, we focus more on the background instead of text regions. Once text regions are detected, they are further processed by two methods to perform text recognition, i.e., ABBYY commercial OCR engine and a state-of-the-art character recognition algorithm. Then, to perform textual cue encoding, bi- and trigrams are formed between the recognized characters by considering the proposed spatial pairwise constraints. Finally, extracted visual and textual cues are combined for fine-grained classification. The proposed method is validated on four publicly available data sets: ICDAR03, ICDAR13, Con-Text, and Flickr-logo. We improve the state-of-the-art end-to-end character recognition by a large margin of 15% on ICDAR03. We show that textual cues are useful in addition to visual cues for fine-grained classification. We show that textual cues are also useful for logo retrieval. Adding textual cues outperforms visual- and textual-only in fine-grained classification (70.7% to 60.3%) and logo retrieval (57.4% to 54.8%).

Evaluation of color spatio-temporal interest points for human action recognition.

This paper considers the recognition of realistic human actions in videos based on spatio-temporal interest points (STIPs). Existing STIP-based action recognition approaches operate on intensity representations of the image data. Because of this, these approaches are sensitive to disturbing photometric phenomena, such as shadows and highlights. In addition, valuable information is neglected by discarding chromaticity from the photometric representation. These issues are addressed by color STIPs. Color STIPs are multichannel reformulations of STIP detectors and descriptors, for which we consider a number of chromatic and invariant representations derived from the opponent color space. Color STIPs are shown to outperform their intensity-based counterparts on the challenging UCF sports, UCF11 and UCF50 action recognition benchmarks by more than 5% on average, where most of the gain is due to the multichannel descriptors. In addition, the results show that color STIPs are currently the single best low-level feature choice for STIP-based approaches to human action recognition.

Robustifying descriptor instability using Fisher vectors.

Many computer vision applications, including image classification, matching, and retrieval use global image representations, such as the Fisher vector, to encode a set of local image patches. To describe these patches, many local descriptors have been designed to be robust against lighting changes and noise. However, local image descriptors are unstable when the underlying image signal is low. Such low-signal patches are sensitive to small image perturbations, which might come e.g., from camera noise or lighting effects. In this paper, we first quantify the relation between the signal strength of a patch and the instability of that patch, and second, we extend the standard Fisher vector framework to explicitly take the descriptor instabilities into account. In comparison to common approaches to dealing with descriptor instabilities, our results show that modeling local descriptor instability is beneficial for object matching, image retrieval, and classification.

Visual word ambiguity.

This paper studies automatic image classification by modeling soft assignment in the popular codebook model. The codebook model describes an image as a bag of discrete visual words selected from a vocabulary, where the frequency distributions of visual words in an image allow classification. One inherent component of the codebook model is the assignment of discrete visual words to continuous image features. Despite the clear mismatch of this hard assignment with the nature of continuous features, the approach has been successfully applied for some years. In this paper, we investigate four types of soft assignment of visual words to image features. We demonstrate that explicitly modeling visual word assignment ambiguity improves classification performance compared to the hard assignment of the traditional codebook model. The traditional codebook model is compared against our method for five well-known data sets: 15 natural scenes, Caltech-101, Caltech-256, and Pascal VOC 2007/2008. We demonstrate that large codebook vocabulary sizes completely deteriorate the performance of the traditional model, whereas the proposed model performs consistently. Moreover, we show that our method profits in high-dimensional feature spaces and reaps higher benefits when increasing the number of image categories.