Full Point Encoding for Local Feature Aggregation in 3-D Point Clouds.

Point cloud processing methods exploit local point features and global context through aggregation which does not explicitly model the internal correlations between local and global features. To address this problem, we propose full point encoding which is applicable to convolution and transformer architectures. Specifically, we propose full point convolution (FuPConv) and full point transformer (FPTransformer) architectures. The key idea is to adaptively learn the weights from local and global geometric connections, where the connections are established through local and global correlation functions, respectively. FuPConv and FPTransformer simultaneously model the local and global geometric relationships as well as their internal correlations, demonstrating strong generalization ability and high performance. FuPConv is incorporated in classical hierarchical network architectures to achieve local and global shape-aware learning. In FPTransformer, we introduce full point position encoding in self-attention, that hierarchically encodes each point position in the global and local receptive field. We also propose a shape-aware downsampling block that takes into account the local shape and the global context. Experimental comparison to existing methods on benchmark datasets shows the efficacy of FuPConv and FPTransformer for semantic segmentation, object detection, classification, and normal estimation tasks. In particular, we achieve state-of-the-art semantic segmentation results of 76.8% mIoU on S3DIS sixfold and 73.1% on S3DIS Area 5. Our code is available at https://github.com/hnuhyuwa/FullPointTransformer.

Region Aware Video Object Segmentation With Deep Motion Modeling.

Current semi-supervised video object segmentation (VOS) methods often employ the entire features of one frame to predict object masks and update memory. This introduces significant redundant computations. To reduce redundancy, we introduce a Region Aware Video Object Segmentation (RAVOS) approach, which predicts regions of interest (ROIs) for efficient object segmentation and memory storage. RAVOS includes a fast object motion tracker to predict object ROIs in the next frame. For efficient segmentation, object features are extracted based on the ROIs, and an object decoder is designed for object-level segmentation. For efficient memory storage, we propose motion path memory to filter out redundant context by memorizing the features within the motion path of objects. In addition to RAVOS, we also propose a large-scale occluded VOS dataset, dubbed OVOS, to benchmark the performance of VOS models under occlusions. Evaluation on DAVIS and YouTube-VOS benchmarks and our new OVOS dataset show that our method achieves state-of-the-art performance with significantly faster inference time, e.g., 86.1 J & F at 42 FPS on DAVIS and 84.4 J & F at 23 FPS on YouTube-VOS. Project page: ravos.netlify.app.

MH6D: Multi-Hypothesis Consistency Learning for Category-Level 6-D Object Pose Estimation.

Six-degree-of-freedom (6DoF) object pose estimation is a crucial task for virtual reality and accurate robotic manipulation. Category-level 6DoF pose estimation has recently become popular as it improves generalization to a complete category of objects. However, current methods focus on data-driven differential learning, which makes them highly dependent on the quality of the real-world labeled data and limits their ability to generalize to unseen objects. To address this problem, we propose multi-hypothesis (MH) consistency learning (MH6D) for category-level 6-D object pose estimation without using real-world training data. MH6D uses a parallel consistency learning structure, alleviating the uncertainty problem of single-shot feature extraction and promoting self-adaptation of domain to reduce the synthetic-to-real domain gap. Specifically, three randomly sampled pose transformations are first performed in parallel on the input point cloud. An attention-guided category-level 6-D pose estimation network with channel attention (CA) and global feature cross-attention (GFCA) modules is then proposed to estimate the three hypothesized 6-D object poses by extracting and fusing the global and local features effectively. Finally, we propose a novel loss function that considers both the process and the final result information allowing MH6D to perform robust consistency learning. We conduct experiments under two different training data settings (i.e., only synthetic data and synthetic and real-world data) to verify the generalization ability of MH6D. Extensive experiments on benchmark datasets demonstrate that MH6D achieves state-of-the-art (SOTA) performance, outperforming most data-driven methods even without using any real-world data. The code is available at https://github.com/CNJianLiu/MH6D.

The expert eye? An inter-rater comparison of elite tennis serve kinematics and performance.

This study examined the reliability of expert tennis coaches/biomechanists to qualitatively assess selected features of the serve with the aid of two-dimensional (2D) video replays. Two expert high-performance coaches rated the serves of 150 male and 150 female players across three different age groups from two different camera viewing angles. Serve performance was rated across 13 variables that represented commonly investigated and coached (serve) mechanics using a 1-7 Likert rating scale. A total of 7800 ratings were performed. The reliability of the experts' ratings was assessed using a Krippendorffs alpha. Strong agreement was shown across all age groups and genders when the experts rated the overall serve score (0.727-0.924), power or speed of the serve (0.720-0.907), rhythm (0.744-0.944), quality of the trunk action (0.775-1.000), leg drive (0.731-0.959) and the likelihood of back injury (0.703-0.934). They encountered greater difficulty in consistently rating shoulder internal rotation speed (0.688-0.717). In high-performance settings, the desire for highly precise measurement and large data sets powered by new technologies, is commonplace but this study revealed that tennis experts, through the use of 2D video, can reliably rate important mechanical features of the game's most important shot, the serve.

A Privacy-Preserving Framework Using Homomorphic Encryption for Smart Metering Systems.

Smart metering systems (SMSs) have been widely used by industrial users and residential customers for purposes such as real-time tracking, outage notification, quality monitoring, load forecasting, etc. However, the consumption data it generates can violate customers' privacy through absence detection or behavior recognition. Homomorphic encryption (HE) has emerged as one of the most promising methods to protect data privacy based on its security guarantees and computability over encrypted data. However, SMSs have various application scenarios in practice. Consequently, we used the concept of trust boundaries to help design HE solutions for privacy protection under these different scenarios of SMSs. This paper proposes a privacy-preserving framework as a systematic privacy protection solution for SMSs by implementing HE with trust boundaries for various SMS scenarios. To show the feasibility of the proposed HE framework, we evaluated its performance on two computation metrics, summation and variance, which are often used for billing, usage predictions, and other related tasks. The security parameter set was chosen to provide a security level of 128 bits. In terms of performance, the aforementioned metrics could be computed in 58,235 ms for summation and 127,423 ms for variance, given a sample size of 100 households. These results indicate that the proposed HE framework can protect customer privacy under varying trust boundary scenarios in SMS. The computational overhead is acceptable from a cost-benefit perspective while ensuring data privacy.

Mesh Convolution With Continuous Filters for 3-D Surface Parsing.

Geometric feature learning for 3-D surfaces is critical for many applications in computer graphics and 3-D vision. However, deep learning currently lags in hierarchical modeling of 3-D surfaces due to the lack of required operations and/or their efficient implementations. In this article, we propose a series of modular operations for effective geometric feature learning from 3-D triangle meshes. These operations include novel mesh convolutions, efficient mesh decimation, and associated mesh (un)poolings. Our mesh convolutions exploit spherical harmonics as orthonormal bases to create continuous convolutional filters. The mesh decimation module is graphics processing unit (GPU)-accelerated and able to process batched meshes on-the-fly, while the (un)pooling operations compute features for upsampled/downsampled meshes. We provide an open-source implementation of these operations, collectively termed Picasso. Picasso supports heterogeneous mesh batching and processing. Leveraging its modular operations, we further contribute a novel hierarchical neural network for perceptual parsing of 3-D surfaces, named PicassoNet ++ . It achieves highly competitive performance for shape analysis and scene segmentation on prominent 3-D benchmarks. The code, data, and trained models are available at https://github.com/EnyaHermite/Picasso.

DualConv: Dual Convolutional Kernels for Lightweight Deep Neural Networks.

Convolutional neural network (CNN) architectures are generally heavy on memory and computational requirements which make them infeasible for embedded systems with limited hardware resources. We propose dual convolutional kernels (DualConv) for constructing lightweight deep neural networks. DualConv combines 3×3 and 1×1 convolutional kernels to process the same input feature map channels simultaneously and exploits the group convolution technique to efficiently arrange convolutional filters. DualConv can be employed in any CNN model such as VGG-16 and ResNet-50 for image classification, you only look once (YOLO) and R-CNN for object detection, or fully convolutional network (FCN) for semantic segmentation. In this work, we extensively test DualConv for classification since these network architectures form the backbone for many other tasks. We also test DualConv for image detection on YOLO-V3. Experimental results show that, combined with our structural innovations, DualConv significantly reduces the computational cost and number of parameters of deep neural networks while surprisingly achieving slightly higher accuracy than the original models in some cases. We use DualConv to further reduce the number of parameters of the lightweight MobileNetV2 by 54% with only 0.68% drop in accuracy on CIFAR-100 dataset. When the number of parameters is not an issue, DualConv increases the accuracy of MobileNetV1 by 4.11% on the same dataset. Furthermore, DualConv significantly improves the YOLO-V3 object detection speed and improves its accuracy by 4.4% on PASCAL visual object classes (VOC) dataset.

Accuracy of mandibular repositioning surgery using new technology: Computer-aided design and manufacturing customized surgical cutting guides and fixation plates.

INTRODUCTION: Recent 3-dimensional technology advancements have resulted in new techniques to improve the accuracy of intraoperative transfer. This study aimed to validate the accuracy of computer-aided design and manufacturing (CAD-CAM) customized surgical cutting guides and fixation plates on mandibular repositioning surgery performed in isolation or combined with simultaneous maxillary repositioning surgery. METHODS: Sixty patients who underwent mandibular advancement surgery by the same surgeon were retrospectively evaluated by 3-dimensional surface-based superimposition. A 3-point coordinate system (x, y, z) was used to identify the linear and angular discrepancies between the planned movements and actual outcomes. Wilcoxon rank sum test was used to compare the outcomes between the mandible-only and the bimaxillary surgery groups with significance at P <0.05. Pearson correlation coefficient compared planned mandible advancement to the outcome from advancement planned. The centroid, which represents the mandible as a single unit, was computed from 3 landmarks, and the discrepancies were evaluated by the root mean square error (RMSE) for clinical significance set at 2 mm for linear discrepancies and 4° for angular discrepancies. RESULTS: There was no statistically significant difference between the planned and actual position of the mandible in either group when considering absolute values of the differences. When considering raw directional data, a statistically significant difference was identified in the y-axis suggesting a tendency for under-advancement of the mandible in the bimaxillary group. The largest translational RMSE for the centroid was 0.77 mm in the sagittal dimension for the bimaxillary surgery group. The largest rotational RMSE for the centroid was 1.25° in the transverse dimension for the bimaxillary surgery group. Our results show that the precision and clinical feasibility of CAD-CAM customized surgical cutting guides and fixation plates on mandibular repositioning surgery is well within clinically acceptable parameters. CONCLUSION: Mandibular repositioning surgery can be performed predictably and accurately with the aid of CAD-CAM customized surgical cutting guides and fixation plates with or without maxillary surgery.

Fully Convolutional Network-Based Self-Supervised Learning for Semantic Segmentation.

Although deep learning has achieved great success in many computer vision tasks, its performance relies on the availability of large datasets with densely annotated samples. Such datasets are difficult and expensive to obtain. In this article, we focus on the problem of learning representation from unlabeled data for semantic segmentation. Inspired by two patch-based methods, we develop a novel self-supervised learning framework by formulating the jigsaw puzzle problem as a patch-wise classification problem and solving it with a fully convolutional network. By learning to solve a jigsaw puzzle comprising 25 patches and transferring the learned features to semantic segmentation task, we achieve a 5.8% point improvement on the Cityscapes dataset over the baseline model initialized from random values. It is noted that we use only about 1/6 training images of Cityscapes in our experiment, which is designed to imitate the real cases where fully annotated images are usually limited to a small number. We also show that our self-supervised learning method can be applied to different datasets and models. In particular, we achieved competitive performance with the state-of-the-art methods on the PASCAL VOC2012 dataset using significantly fewer time costs on pretraining.

An investigation of a novel broad autism phenotype: increased facial masculinity among parents of children on the autism spectrum.

The broad autism phenotype commonly refers to sub-clinical levels of autistic-like behaviour and cognition presented in biological relatives of autistic people. In a recent study, we reported findings suggesting that the broad autism phenotype may also be expressed in facial morphology, specifically increased facial masculinity. Increased facial masculinity has been reported among autistic children, as well as their non-autistic siblings. The present study builds on our previous findings by investigating the presence of increased facial masculinity among non-autistic parents of autistic children. Using a previously established method, a 'facial masculinity score' and several facial distances were calculated for each three-dimensional facial image of 192 parents of autistic children (58 males, 134 females) and 163 age-matched parents of non-autistic children (50 males, 113 females). While controlling for facial area and age, significantly higher masculinity scores and larger (more masculine) facial distances were observed in parents of autistic children relative to the comparison group, with effect sizes ranging from small to medium (0.16 ≤ d ≤ .41), regardless of sex. These findings add to an accumulating evidence base that the broad autism phenotype is expressed in physical characteristics and suggest that both maternal and paternal pathways are implicated in masculinized facial morphology.

Autoencoded deep features for semi-automatic, weakly supervised physiological signal labelling.

BACKGROUND AND AIMS: Machine Learning is transforming data processing in medical research and clinical practice. Missing data labels are a common limitation to training Machine Learning models. To overcome missing labels in a large dataset of microneurography recordings, a novel autoencoder based semi-supervised, iterative group-labelling methodology was developed. METHODS: Autoencoders were systematically optimised to extract features from a dataset of 478621 signal excerpts from human microneurography recordings. Selected features were clusters with k-means and randomly selected representations of the corresponding original signals labelled as valid or non-valid muscle sympathetic nerve activity (MSNA) bursts in an iterative, purifying procedure by an expert rater. A deep neural network was trained based on the fully labelled dataset. RESULTS: Three autoencoders, two based on fully connected neural networks and one based on convolutional neural network, were chosen for feature learning. Iterative clustering followed by labelling of complete clusters resulted in all 478621 signal peak excerpts being labelled as valid or non-valid within 13 iterations. Neural networks trained with the labelled dataset achieved, in a cross validation step with a testing dataset not included in training, on average 93.13% accuracy and 91% area under the receiver operating curve (AUC ROC). DISCUSSION: The described labelling procedure enabled efficient labelling of a large dataset of physiological signal based on expert ratings. The procedure based on autoencoders may be broadly applicable to a wide range of datasets without labels that require expert input and may be utilised for Machine Learning applications if weak-labels were available.

Adversarial Attack on Skeleton-Based Human Action Recognition.

Deep learning models achieve impressive performance for skeleton-based human action recognition. Graph convolutional networks (GCNs) are particularly suitable for this task due to the graph-structured nature of skeleton data. However, the robustness of these models to adversarial attacks remains largely unexplored due to their complex spatiotemporal nature that must represent sparse and discrete skeleton joints. This work presents the first adversarial attack on skeleton-based action recognition with GCNs. The proposed targeted attack, termed constrained iterative attack for skeleton actions (CIASA), perturbs joint locations in an action sequence such that the resulting adversarial sequence preserves the temporal coherence, spatial integrity, and the anthropomorphic plausibility of the skeletons. CIASA achieves this feat by satisfying multiple physical constraints and employing spatial skeleton realignments for the perturbed skeletons along with regularization of the adversarial skeletons with generative networks. We also explore the possibility of semantically imperceptible localized attacks with CIASA and succeed in fooling the state-of-the-art skeleton action recognition models with high confidence. CIASA perturbations show high transferability in black-box settings. We also show that the perturbed skeleton sequences are able to induce adversarial behavior in the RGB videos created with computer graphics. A comprehensive evaluation with NTU and Kinetics data sets ascertains the effectiveness of CIASA for graph-based skeleton action recognition and reveals the imminent threat to the spatiotemporal deep learning tasks in general.

Attack to Fool and Explain Deep Networks.

Deep visual models are susceptible to adversarial perturbations to inputs. Although these signals are carefully crafted, they still appear noise-like patterns to humans. This observation has led to the argument that deep visual representation is misaligned with human perception. We counter-argue by providing evidence of human-meaningful patterns in adversarial perturbations. We first propose an attack that fools a network to confuse a whole category of objects (source class) with a target label. Our attack also limits the unintended fooling by samples from non-sources classes, thereby circumscribing human-defined semantic notions for network fooling. We show that the proposed attack not only leads to the emergence of regular geometric patterns in the perturbations, but also reveals insightful information about the decision boundaries of deep models. Exploring this phenomenon further, we alter the 'adversarial' objective of our attack to use it as a tool to 'explain' deep visual representation. We show that by careful channeling and projection of the perturbations computed by our method, we can visualize a model's understanding of human-defined semantic notions. Finally, we exploit the explanability properties of our perturbations to perform image generation, inpainting and interactive image manipulation by attacking adversarialy robust 'classifiers'. In all, our major contribution is a novel pragmatic adversarial attack that is subsequently transformed into a tool to interpret the visual models. The article also makes secondary contributions in terms of establishing the utility of our attack beyond the adversarial objective with multiple interesting applications.

Fast ORB-SLAM Without Keypoint Descriptors.

Indirect methods for visual SLAM are gaining popularity due to their robustness to environmental variations. ORB-SLAM2 (Mur-Artal and Tardós, 2017) is a benchmark method in this domain, however, it consumes significant time for computing descriptors that never get reused unless a frame is selected as a keyframe. To overcome these problems, we present FastORB-SLAM which is light-weight and efficient as it tracks keypoints between adjacent frames without computing descriptors. To achieve this, a two stage descriptor-independent keypoint matching method is proposed based on sparse optical flow. In the first stage, we predict initial keypoint correspondences via a simple but effective motion model and then robustly establish the correspondences via pyramid-based sparse optical flow tracking. In the second stage, we leverage the constraints of the motion smoothness and epipolar geometry to refine the correspondences. In particular, our method computes descriptors only for keyframes. We test FastORB-SLAM on TUM and ICL-NUIM RGB-D datasets and compare its accuracy and efficiency to nine existing RGB-D SLAM methods. Qualitative and quantitative results show that our method achieves state-of-the-art accuracy and is about twice as fast as the ORB-SLAM2.

Automatic data extraction from 24 hour blood pressure measurement reports of a large multicenter clinical trial.

BACKGROUND AND OBJECTIVES: Ambulatory blood pressure monitoring (ABPM) is usually reported in descriptive values such as circadian averages and standard deviations. Making use of the original, individual blood pressure measurements may be advantageous, particularly for research purposes, as this increases the flexibility of the analytical process, enables alternative statistical analyses and provide novel insights. Here we describe the development of a new multistep, hierarchical data extraction algorithm to collect raw data from .pdf reports and text files as part of a large multi-center clinical study. METHODS: Original reports were saved in a nested file system, from which they were automatically extracted, read and saved into databases with custom made programs written in Python 3. Data were further processed, cleaned and relevant descriptive statistics such as averages and standard deviations calculated according to a variety of definitions of day- and night-time. Additionally, data control mechanisms for manual review of the data and programmatic auto-detection of extraction errors was implemented as part of the project. RESULTS: The developed algorithm extracted 97% of the data automatically, the missing data consisted mostly of reports that were saved incorrectly or not formatted in the specified way. Manual checks comparing samples of the extracted data to original reports indicated a high level of accuracy of the extracted data, no errors introduced due to flaws in the extraction software were detected in the extracted dataset. CONCLUSIONS: The developed multistep, hierarchical data extraction algorithm facilitated collection from different file formats and paired with database cleaning and data processing steps led to an effective and accurate assembly of raw ABPM data for further and adjustable analyses. Manual work was minimized while data quality was ensured with standardized, reproducible procedures.

K-means panning - Developing a new standard in automated MSNA signal recognition with a weakly supervised learning approach.

BACKGROUND: Accessibility of labelled datasets is often a key limitation for the application of Machine Learning in clinical research. A novel semi-automated weak-labelling approach based on unsupervised clustering was developed to classify a large dataset of microneurography signals and subsequently used to train a Neural Network to reproduce the labelling process. METHODS: Clusters of microneurography signals were created with k-means and then labelled in terms of the validity of the signals contained in each cluster. Only purely positive or negative clusters were labelled, whereas clusters with mixed content were passed on to the next iteration of the algorithm to undergo another cycle of unsupervised clustering and labelling of the clusters. After several iterations of this process, only pure labelled clusters remained which were used to train a Deep Neural Network. RESULTS: Overall, 334,548 individual signal peaks form the integrated data were extracted and more than 99.99% of the data was labelled in six iterations of this novel application of weak labelling with the help of a domain expert. A Deep Neural Network trained based on this dataset achieved consistent accuracies above 95%. DISCUSSION: Data extraction and the novel iterative approach of labelling unsupervised clusters enabled creation of a large, labelled dataset combining unsupervised learning and expert ratings of signal-peaks on cluster basis in a time effective manner. Further research is needed to validate the methodology and employ it on other types of physiologic data for which it may enable efficient generation of large labelled datasets.

Facial asymmetry in parents of children on the autism spectrum.

Greater facial asymmetry has been consistently found in children with autism spectrum disorder (ASD) relative to children without ASD. There is substantial evidence that both facial structure and the recurrence of ASD diagnosis are highly heritable within a nuclear family. Furthermore, sub-clinical levels of autistic-like behavioural characteristics have also been reported in first-degree relatives of individuals with ASD, commonly known as the 'broad autism phenotype'. Therefore, the aim of the current study was to examine whether a broad autism phenotype expresses as facial asymmetry among 192 biological parents of autistic individuals (134 mothers) compared to those of 163 age-matched adults without a family history of ASD (113 females). Using dense surface-modelling techniques on three dimensional facial images, we found evidence for greater facial asymmetry in parents of autistic individuals compared to age-matched adults in the comparison group (p = 0.046, d = 0.21 [0.002, 0.42]). Considering previous findings and the current results, we conclude that facial asymmetry expressed in the facial morphology of autistic children may be related to heritability factors. LAY ABSTRACT: In a previous study, we showed that autistic children presented with greater facial asymmetry than non-autistic children. In the current study, we examined the amount of facial asymmetry shown on three-dimensional facial images of 192 parents of autistic children compared to a control group consisting of 163 similarly aged adults with no known history of autism. Although parents did show greater levels of facial asymmetry than those in the control group, this effect is statistically small. We concluded that the facial asymmetry previously found in autistic children may be related to genetic factors.

Machine learning powered tools for automated analysis of muscle sympathetic nerve activity recordings.

Automated analysis and quantification of physiological signals in clinical practice and medical research can reduce manual labor, increase efficiency, and provide more objective, reproducible results. To build a novel platform for the analysis of muscle sympathetic nerve activity (MSNA), we employed state-of-the-art data processing and machine learning applications. Data processing methods for integrated MSNA recordings were developed to evaluate signals regarding the overall quality of the signal, the validity of individual signal peaks regarding the potential to be MSNA bursts and the timing of their occurrence. An overall probability score was derived from this flexible platform to evaluate each individual signal peak automatically. Overall, three deep neural networks were designed and trained to validate individual signal peaks randomly sampled from recordings representing only electrical noise and valid microneurography recordings. A novel data processing method for the whole signal was developed to differentiate between periods of valid MSNA signal recordings and periods in which the signal was not available or lost due to involuntary movement of the recording electrode. A probabilistic model for timing of the signal bursts was implemented as part of the system. Machine Learning algorithms and data processing tools were implemented to replicate the complex decision-making process of manual MSNA analysis. Validation of manual MSNA analysis including intra- and inter-rater validity and a comparison with automated MSNA tools is required. The developed toolbox for automated MSNA analysis can be extended in a flexible way to include algorithms based on other datasets.

A Comparison of Three Neural Network Approaches for Estimating Joint Angles and Moments from Inertial Measurement Units.

The application of artificial intelligence techniques to wearable sensor data may facilitate accurate analysis outside of controlled laboratory settings-the holy grail for gait clinicians and sports scientists looking to bridge the lab to field divide. Using these techniques, parameters that are difficult to directly measure in-the-wild, may be predicted using surrogate lower resolution inputs. One example is the prediction of joint kinematics and kinetics based on inputs from inertial measurement unit (IMU) sensors. Despite increased research, there is a paucity of information examining the most suitable artificial neural network (ANN) for predicting gait kinematics and kinetics from IMUs. This paper compares the performance of three commonly employed ANNs used to predict gait kinematics and kinetics: multilayer perceptron (MLP); long short-term memory (LSTM); and convolutional neural networks (CNN). Overall high correlations between ground truth and predicted kinematic and kinetic data were found across all investigated ANNs. However, the optimal ANN should be based on the prediction task and the intended use-case application. For the prediction of joint angles, CNNs appear favourable, however these ANNs do not show an advantage over an MLP network for the prediction of joint moments. If real-time joint angle and joint moment prediction is desirable an LSTM network should be utilised.