Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks.

Modeling and simulation of complex non-linear systems are essential in physics, engineering, and signal processing. Neural networks are widely regarded for such tasks due to their ability to learn complex representations from data. Training deep neural networks traditionally requires large amounts of data, which may not always be readily available for such systems. Contrarily, there is a large amount of domain knowledge in the form of mathematical models for the physics/behavior of such systems. A new class of neural networks called Physics-Informed Neural Networks (PINNs) has gained much attention recently as a paradigm for combining physics into neural networks. They have become a powerful tool for solving forward and inverse problems involving differential equations. A general framework of a PINN consists of a multi-layer perceptron that learns the solution of the partial differential equation (PDE) along with its boundary/initial conditions by minimizing a multi-objective loss function. This is formed by the sum of individual loss terms that penalize the output at different collocation points based on the differential equation and initial and boundary conditions. However, multiple loss terms arising from PDE residual and boundary conditions in PINNs pose a challenge in optimizing the overall loss function. This often leads to training failures and inaccurate results. We propose advanced gradient statistics-based weighting schemes for PINNs to address this challenge. These schemes utilize backpropagated gradient statistics of individual loss terms to appropriately scale and assign weights to each term, ensuring balanced training and meaningful solutions. In addition to the existing gradient statistics-based weighting schemes, we introduce kurtosis-standard deviation-based and combined mean and standard deviation-based schemes for approximating solutions of PDEs using PINNs. We provide a qualitative and quantitative comparison of these weighting schemes on 2D Poisson's and Klein-Gordon's equations, highlighting their effectiveness in improving PINN performance.

Limb, joint and pelvic kinematic control in the quail coping with steps upwards and downwards.

Small cursorial birds display remarkable walking skills and can negotiate complex and unstructured terrains with ease. The neuromechanical control strategies necessary to adapt to these challenging terrains are still not well understood. Here, we analyzed the 2D- and 3D pelvic and leg kinematic strategies employed by the common quail to negotiate visible steps (upwards and downwards) of about 10%, and 50% of their leg length. We used biplanar fluoroscopy to accurately describe joint positions in three dimensions and performed semi-automatic landmark localization using deep learning. Quails negotiated the vertical obstacles without major problems and rapidly regained steady-state locomotion. When coping with step upwards, the quail mostly adapted the trailing limb to permit the leading leg to step on the elevated substrate similarly as it did during level locomotion. When negotiated steps downwards, both legs showed significant adaptations. For those small and moderate step heights that did not induce aerial running, the quail kept the kinematic pattern of the distal joints largely unchanged during uneven locomotion, and most changes occurred in proximal joints. The hip regulated leg length, while the distal joints maintained the spring-damped limb patterns. However, to negotiate the largest visible steps, more dramatic kinematic alterations were observed. There all joints contributed to leg lengthening/shortening in the trailing leg, and both the trailing and leading legs stepped more vertically and less abducted. In addition, locomotion speed was decreased. We hypothesize a shift from a dynamic walking program to more goal-directed motions that might be focused on maximizing safety.

Automatic Objective Severity Grading of Peripheral Facial Palsy Using 3D Radial Curves Extracted from Point Clouds.

Peripheral facial palsy is an illness in which a one-sided ipsilateral paralysis of the facial muscles occurs due to nerve damage. Medical experts utilize visual severity grading methods to estimate this damage. Our algorithm-based method provides an objective grading using 3D point clouds. We extract from static 3D recordings facial radial curves to measure volumetric differences between both sides of the face. We analyze five patients with chronic complete peripheral facial palsy to evaluate our method by comparing changes over several recording sessions. We show that our proposed method allows an objective assessment official palsy.

Generative adversarial networks for biomedical time series forecasting and imputation.

In the present systematic review we identified and summarised current research activities in the field of time series forecasting and imputation with the help of generative adversarial networks (GANs). We differentiate between imputation which describes the filling of missing values at intermediate steps and forecasting defining the prediction of future values. Especially the utilisation of such methods in the biomedical domain was to be investigated. To this end, 1057 publications were identified with the help of PubMed, Web of Science and Scopus. All studies that describe the use of GANs for the imputation/forecasting of time series were included irrespective of the application domain. Finally, 33 records were identified as eligible and grouped according to the topologies, losses, inputs and outputs of the presented GANs. In combination with a summary of all described application domains, this grouping served as a basis for analysing the peculiarities of the method in the biomedical context. Due to the broad spectrum of biomedical research, nearly all recognised methodologies are also applied in this domain. We could not identify any approach that proved itself superior in the biomedical area. Although GANs were initially designed to work in the image domain, many publications show that they are capable of imputing/forecasting non-visual time series.

Do We Train on Test Data? Purging CIFAR of Near-Duplicates.

The CIFAR-10 and CIFAR-100 datasets are two of the most heavily benchmarked datasets in computer vision and are often used to evaluate novel methods and model architectures in the field of deep learning. However, we find that 3.3% and 10% of the images from the test sets of these datasets have duplicates in the training set. These duplicates are easily recognizable by memorization and may, hence, bias the comparison of image recognition techniques regarding their generalization capability. To eliminate this bias, we provide the "fair CIFAR" (ciFAIR) dataset, where we replaced all duplicates in the test sets with new images sampled from the same domain. The training set remains unchanged, in order not to invalidate pre-trained models. We then re-evaluate the classification performance of various popular state-of-the-art CNN architectures on these new test sets to investigate whether recent research has overfitted to memorizing data instead of learning abstract concepts. We find a significant drop in classification accuracy of between 9% and 14% relative to the original performance on the duplicate-free test set. We make both the ciFAIR dataset and pre-trained models publicly available and furthermore maintain a leaderboard for tracking the state of the art.

The Whole Is More Than Its Parts? From Explicit to Implicit Pose Normalization.

Fine-grained classification describes the automated recognition of visually similar object categories like birds species. Previous works were usually based on explicit pose normalization, i.e., the detection and description of object parts. However, recent models based on a final global average or bilinear pooling have achieved a comparable accuracy without this concept. In this paper, we analyze the advantages of these approaches over generic CNNs and explicit pose normalization approaches. We also show how they can achieve an implicit normalization of the object pose. A novel visualization technique called activation flow is introduced to investigate limitations in pose handling in traditional CNNs like AlexNet and VGG. Afterward, we present and compare the explicit pose normalization approach neural activation constellations and a generalized framework for the final global average and bilinear pooling called α-pooling. We observe that the latter often achieves a higher accuracy improving common CNN models by up to 22.9 percent, but lacks the interpretability of the explicit approaches. We present a visualization approach for understanding and analyzing predictions of the model to address this issue. Furthermore, we show that our approaches for fine-grained recognition are beneficial for other fields like action recognition.

Automated objective and marker-free facial grading using photographs of patients with facial palsy.

PURPOSE: An automated, objective, fast and simple classification system for the grading of facial palsy (FP) is lacking. METHODS: An observational single center study was performed. 4572 photographs of 233 patients with unilateral peripheral FP were subjectively rated and automatically analyzed applying a machine learning approach including Supervised Descent Method. This allowed an automated grading of all photographs according to House-Brackmann grading scale (HB), Sunnybrook grading system (SB), and Stennert index (SI). RESULTS: Median time to first assessment was 6 days after onset. At first examination, the median objective HB, total SB, and total SI were grade 3, 45, and 5, respectively. The best correlation between subjective and objective grading was seen for SB and SI movement score (r = 0.746; r = 0.732, respectively). No agreement was found between subjective and objective HB grading [Test for symmetry 80.61, df = 15, p < 0.001, weighted kappa = - 0.0105; 95% confidence interval (CI) = - 0.0542 to 0.0331; p = 0.6541]. Also no agreement was found between subjective and objective total SI (test for symmetry 166.37, df = 55, p < 0.001) although there was a nonzero weighted kappa = 0.2670; CI 0.2154-0.3186; p < 0.0001). Based on a multinomial logistic regression the probability for higher scores was higher for subjective compared to objective SI (OR 1.608; CI 1.202-2.150; p = 0.0014). The best agreement was seen between subjective and objective SB (ICC = 0.34645). CONCLUSIONS: Automated Sunnybrook grading delivered with fair agreement fast and objective global and regional data on facial motor function for use in clinical routine and clinical trials.

Deep learning and process understanding for data-driven Earth system science.

Machine learning approaches are increasingly used to extract patterns and insights from the ever-increasing stream of geospatial data, but current approaches may not be optimal when system behaviour is dominated by spatial or temporal context. Here, rather than amending classical machine learning, we argue that these contextual cues should be used as part of deep learning (an approach that is able to extract spatio-temporal features automatically) to gain further process understanding of Earth system science problems, improving the predictive ability of seasonal forecasting and modelling of long-range spatial connections across multiple timescales, for example. The next step will be a hybrid modelling approach, coupling physical process models with the versatility of data-driven machine learning.

Detecting Regions of Maximal Divergence for Spatio-Temporal Anomaly Detection.

Automatic detection of anomalies in space- and time-varying measurements is an important tool in several fields, e.g., fraud detection, climate analysis, or healthcare monitoring. We present an algorithm for detecting anomalous regions in multivariate spatio-temporal time-series, which allows for spotting the interesting parts in large amounts of data, including video and text data. In opposition to existing techniques for detecting isolated anomalous data points, we propose the "Maximally Divergent Intervals" (MDI) framework for unsupervised detection of coherent spatial regions and time intervals characterized by a high Kullback-Leibler divergence compared with all other data given. In this regard, we define an unbiased Kullback-Leibler divergence that allows for ranking regions of different size and show how to enable the algorithm to run on large-scale data sets in reasonable time using an interval proposal technique. Experiments on both synthetic and real data from various domains, such as climate analysis, video surveillance, and text forensics, demonstrate that our method is widely applicable and a valuable tool for finding interesting events in different types of data.

Facial motor and non-motor disabilities in patients with central facial paresis: a prospective cohort study.

Although central facial paresis (CFP) is a major symptom of stroke, there is a lack of studies on the motor and non-motor disabilities in stroke patients. A prospective cohort study was performed at admission for inpatient rehabilitation and discharge of post-stroke phase of 112 patients (44% female, median age: 64 years, median Barthel index: 70) with CFP. Motor function was evaluated using House-Brackmann grading, Sunnybrook grading and Stennert Index. Automated action unit (AU) analysis was performed to analyze mimic function in detail. Non-motor function was assessed using the Facial Disability Index (FDI) and the Facial Clinimetric Evaluation (FaCE). Median interval from stroke to rehabilitation was 21 days. Rehabilitation lasted 20 days. House-Brackmann grading was ≥ grade III for 79% at admission. AU activation in the lower face was significantly lower in patients with right hemispheric infarction compared to left hemispheric infarction (all p < 0.05). Median total FDI and FaCE score were 46.5 and 69, respectively. Facial grading and FDI/FaCE scores improved during inpatient rehabilitation (all p < 0.05). There was a significant increase of the activation of AU12 (Zygomaticus major muscle), AU13 (Levator anguli oris muscle), and AU24 (Orbicularis oris muscle) during inpatient rehabilitation (all p < 0.05). Multivariate analysis revealed that activation of AU10 (Levator labii superioris), AU12, AU17 (Depressor labii), and AU 38 (Nasalis) were independent predictors for better quality of life. These results demonstrate that CFP has a significant impact on patient's quality of life. Therapy of CFP with focus on specific AUs should be part of post-stroke rehabilitation.

Fully convolutional networks in multimodal nonlinear microscopy images for automated detection of head and neck carcinoma: Pilot study.

BACKGROUND: A fully convolutional neural networks (FCN)-based automated image analysis algorithm to discriminate between head and neck cancer and noncancerous epithelium based on nonlinear microscopic images was developed. METHODS: Head and neck cancer sections were used for standard histopathology and co-registered with multimodal images from the same sections using the combination of coherent anti-Stokes Raman scattering, two-photon excited fluorescence, and second harmonic generation microscopy. The images analyzed with semantic segmentation using a FCN for four classes: cancer, normal epithelium, background, and other tissue types. RESULTS: A total of 114 images of 12 patients were analyzed. Using a patch score aggregation, the average recognition rate and an overall recognition rate or the four classes were 88.9% and 86.7%, respectively. A total of 113 seconds were needed to process a whole-slice image in the dataset. CONCLUSION: Multimodal nonlinear microscopy in combination with automated image analysis using FCN seems to be a promising technique for objective differentiation between head and neck cancer and noncancerous epithelium.

Reliability of grading of facial palsy using a video tutorial with synchronous video recording.

OBJECTIVE: To determine the intrarater, interrater, and retest reliability of facial nerve grading of patients with facial palsy (FP) using standardized videos recorded synchronously during a self-explanatory patient video tutorial. STUDY DESIGN: Prospective, observational study. METHODS: The automated videos from 10 patients with varying degrees of FP (5 acute, 5 chronic FP) and videos without tutorial from eight patients (all chronic FP) were rated by five novices and five experts according to the House-Brackmann grading system (HB), the Sunnybrook Grading System (SB), and the Facial Nerve Grading System 2.0 (FNGS 2.0). RESULTS: Intrarater reliability for the three grading systems was very high using the automated videos (intraclass correlation coefficient [ICC]; SB: ICC = 0.967; FNGS 2.0: ICC = 0.931; HB: ICC = 0.931). Interrater reliability was also high (SB: ICC = 0.921; FNGS 2.0: ICC = 0.837; HB: ICC = 0.736), but for HB Fleiss kappa (0.214) and Kendell W (0.231) was low. The interrater reliability was not different between novices and experts. Retest reliability was very high (SB: novices ICC = 0.979; experts ICC = 0.964; FNGS 2.0: novices ICC = 0.979; experts ICC = 0.969). The reliability of grading of chronic FP with SB was higher using automated videos with tutorial (ICC = 0.845) than without tutorial (ICC = 0.538). CONCLUSION: The reliability of the grading using the automated videos is excellent, especially for the SB grading. We recommend using this automated video tool regularly in clinical routine and for clinical studies. LEVEL OF EVIDENCE: 4 xsLaryngoscope, 129:2274-2279, 2019.

HER2 challenge contest: a detailed assessment of automated HER2 scoring algorithms in whole slide images of breast cancer tissues.

AIMS: Evaluating expression of the human epidermal growth factor receptor 2 (HER2) by visual examination of immunohistochemistry (IHC) on invasive breast cancer (BCa) is a key part of the diagnostic assessment of BCa due to its recognized importance as a predictive and prognostic marker in clinical practice. However, visual scoring of HER2 is subjective, and consequently prone to interobserver variability. Given the prognostic and therapeutic implications of HER2 scoring, a more objective method is required. In this paper, we report on a recent automated HER2 scoring contest, held in conjunction with the annual PathSoc meeting held in Nottingham in June 2016, aimed at systematically comparing and advancing the state-of-the-art artificial intelligence (AI)-based automated methods for HER2 scoring. METHODS AND RESULTS: The contest data set comprised digitized whole slide images (WSI) of sections from 86 cases of invasive breast carcinoma stained with both haematoxylin and eosin (H&E) and IHC for HER2. The contesting algorithms predicted scores of the IHC slides automatically for an unseen subset of the data set and the predicted scores were compared with the 'ground truth' (a consensus score from at least two experts). We also report on a simple 'Man versus Machine' contest for the scoring of HER2 and show that the automated methods could beat the pathology experts on this contest data set. CONCLUSIONS: This paper presents a benchmark for comparing the performance of automated algorithms for scoring of HER2. It also demonstrates the enormous potential of automated algorithms in assisting the pathologist with objective IHC scoring.

Automatic Classification of Cancerous Tissue in Laserendomicroscopy Images of the Oral Cavity using Deep Learning.

Oral Squamous Cell Carcinoma (OSCC) is a common type of cancer of the oral epithelium. Despite their high impact on mortality, sufficient screening methods for early diagnosis of OSCC often lack accuracy and thus OSCCs are mostly diagnosed at a late stage. Early detection and accurate outline estimation of OSCCs would lead to a better curative outcome and a reduction in recurrence rates after surgical treatment. Confocal Laser Endomicroscopy (CLE) records sub-surface micro-anatomical images for in vivo cell structure analysis. Recent CLE studies showed great prospects for a reliable, real-time ultrastructural imaging of OSCC in situ. We present and evaluate a novel automatic approach for OSCC diagnosis using deep learning technologies on CLE images. The method is compared against textural feature-based machine learning approaches that represent the current state of the art. For this work, CLE image sequences (7894 images) from patients diagnosed with OSCC were obtained from 4 specific locations in the oral cavity, including the OSCC lesion. The present approach is found to outperform the state of the art in CLE image recognition with an area under the curve (AUC) of 0.96 and a mean accuracy of 88.3% (sensitivity 86.6%, specificity 90%).

[Video Instruction for Synchronous Video Recording of Mimic Movement of Patients with Facial Palsy]. / Patienten-Instruktionsvideo mit synchroner Videoaufnahme von Gesichtsbewegungen bei Fazialisparese.

Background Photografy and video are necessary to record the severity of a facial palsy or to allow offline grading with a grading system. There is no international standard for the video recording urgently needed to allow a standardized comparison of different patient cohorts. Methods A video instruction was developed. The instruction was shown to the patient and presents several mimic movements. At the same time the patient is recorded while repeating the presented movement using commercial hardware. Facial movements were selected in such a way that it was afterwards possible to evaluate the recordings with standard grading systems (House-Brackmann, Sunnybrook, Stennert, Yanagihara) or even with (semi)automatic software. For quality control, the patients evaluated the instruction using a questionnaire. Results The video instruction takes 11 min and 05 and is divided in 3 parts: 1) Explanation of the procedure; 2) Foreplay and recreating of the facial movements; 3) Repeating of sentences to analyze the communication skills. So far 13 healthy subjects and 10 patients with acute or chronic facial palsy were recorded. All recordings could be assessed by the above mentioned grading systems. The instruction was rated as well explaining and easy to follow by healthy persons and patients. Discussion There is now a video instruction available for standardized recording of facial movement. This instruction is recommended for use in clinical routine and in clinical trials. This will allow a standardized comparison of patients within Germany and international patient cohorts.

An automated whisker tracking tool for the rat facial nerve injury paradigm.

BACKGROUND: The two-dimensional videographic analysis of vibrissal movements in behaving rodents has become a standard method to estimate the degree of functional impairment and recovery after facial nerve injuries quantitatively. The main limitation of the method is the time consuming, uneconomic process of manually tracking the vibrissae in video sequences. NEW METHOD: We developed a novel tool allowing automated detection of untagged vibrissae (two on each side of the snout). To compare the new method with the standard manual tracking approach, we used videos of unrestrained rats with unilateral section and immediate suture of the facial nerve performed two months earlier. RESULTS: Measurement agreement analyses showed that the two methods are equivalent for both "normal" high-amplitude vibrissal movements (non-operated side) and low-amplitude whisking (reinnervated side). Spectral analysis revealed a significant deviation in the power spectra on the control and injured side, indicating that bilaterally coordinated whisker movements are not present two months after surgery. COMPARISON WITH EXISTING METHOD(S): The novel method yields results equal to those of the manual tracking approach. An advantage of our tool is the possibility to significantly increase sample size without additional labor cost. CONCLUSIONS: The novel tool can increase the efficacy and spectrum of functional measures used in facial nerve regeneration research.

Fully automated tracking of cardiac structures using radiopaque markers and high-frequency videofluoroscopy in an in vivo ovine model: from three-dimensional marker coordinates to quantitative analyses.

PURPOSE: Recently, algorithms were developed to track radiopaque markers in the heart fully automated. However, the methodology did not allow to assign the exact anatomical location to each marker. In this case study we describe the steps from the generation of three-dimensional marker coordinates to quantitative data analyses in an in vivo ovine model. METHODS: In one adult sheep, twenty silver balls were sutured to the right side of the heart: 10 to the tricuspid annulus, one to the anterior tricuspid leaflet and nine to the epicardial surface of the right ventricle. In addition, 13 cylindrical tantalum markers were implanted into the left ventricle. Data were acquired with a biplanar X-ray acquisition system (Neurostar R, Siemens AG, 500 Hz). Radiopaque marker coordinates were determined fully automated using novel tracking algorithms. RESULTS: The anatomical marker locations were identified using a 3-dimensional model of a single frame containing all tracked markers. First, cylindrical markers were manually separated from spherical markers, thus allowing to distinguish right from left heart markers. The fast moving leaflet marker was identified by using video loops constructed of all recorded frames. Rotation of the 3-dimensional model allowed the identification of the precise anatomical position for each marker. Data sets were then analyzed quantitatively using customized software. CONCLUSIONS: The method presented in this case study allowed quantitative data analyses of radiopaque cardiac markers that were tracked fully automated with high temporal resolution. However, marker identification still requires substantial manual work. Future improvements including the implication of marker identification algorithms and data analysis software could allow almost real-time quantitative analyses of distinct cardiac structures with high temporal and spatial resolution.

Automated analysis of confocal laser endomicroscopy images to detect head and neck cancer.

BACKGROUND: The purpose of this study was to develop an automated image analysis algorithm to discriminate between head and neck cancer and nonneoplastic epithelium in confocal laser endomicroscopy (CLE) images. METHODS: CLE was applied to image head and neck cancer epithelium in vivo. Histopathologic diagnosis from biopsies was used to classify the CLE images offline as cancer or noncancer tissue. The classified images were used to train automated software based on distance map histograms. The performance of the final algorithm was confirmed by "leave 2 patients out" cross-validation and area under the curve (AUC)/receiver operating characteristic (ROC) analysis. RESULTS: Ninety-two CLE videos and 92 biopsies were analyzed from 12 patients. One hundred two frames of classified neoplastic tissue and 52 frames of nonneoplastic tissue were used for cross-validation of the developed algorithm. AUC varied from 0.52 to 0.92. CONCLUSION: The proposed software allows an objective classification of CLE images of head and neck cancer and adjacent nonneoplastic epithelium. © 2015 Wiley Periodicals, Inc. Head Neck 38: E1419-E1426, 2016.

Novel computer vision algorithm for the reliable analysis of organelle morphology in whole cell 3D images--A pilot study for the quantitative evaluation of mitochondrial fragmentation in amyotrophic lateral sclerosis.

The function of intact organelles, whether mitochondria, Golgi apparatus or endoplasmic reticulum (ER), relies on their proper morphological organization. It is recognized that disturbances of organelle morphology are early events in disease manifestation, but reliable and quantitative detection of organelle morphology is difficult and time-consuming. Here we present a novel computer vision algorithm for the assessment of organelle morphology in whole cell 3D images. The algorithm allows the numerical and quantitative description of organelle structures, including total number and length of segments, cell and nucleus area/volume as well as novel texture parameters like lacunarity and fractal dimension. Applying the algorithm we performed a pilot study in cultured motor neurons from transgenic G93A hSOD1 mice, a model of human familial amyotrophic lateral sclerosis. In the presence of the mutated SOD1 and upon excitotoxic treatment with kainate we demonstrate a clear fragmentation of the mitochondrial network, with an increase in the number of mitochondrial segments and a reduction in the length of mitochondria. Histogram analyses show a reduced number of tubular mitochondria and an increased number of small mitochondrial segments. The computer vision algorithm for the evaluation of organelle morphology allows an objective assessment of disease-related organelle phenotypes with greatly reduced examiner bias and will aid the evaluation of novel therapeutic strategies on a cellular level.

Mixed gaits in small avian terrestrial locomotion.

Scientists have historically categorized gaits discretely (e.g. regular gaits such as walking, running). However, previous results suggest that animals such as birds might mix or regularly or stochastically switch between gaits while maintaining a steady locomotor speed. Here, we combined a novel and completely automated large-scale study (over one million frames) on motions of the center of mass in several bird species (quail, oystercatcher, northern lapwing, pigeon, and avocet) with numerical simulations. The birds studied do not strictly prefer walking mechanics at lower speeds or running mechanics at higher speeds. Moreover, our results clearly display that the birds in our study employ mixed gaits (such as one step walking followed by one step using running mechanics) more often than walking and, surprisingly, maybe as often as grounded running. Using a bio-inspired model based on parameters obtained from real quails, we found two types of stable mixed gaits. In the first, both legs exhibit different gait mechanics, whereas in the second, legs gradually alternate from one gait mechanics into the other. Interestingly, mixed gaits parameters mostly overlap those of grounded running. Thus, perturbations or changes in the state induce a switch from grounded running to mixed gaits or vice versa.