Deep learning in attosecond metrology.

Time-resolved photoelectron spectroscopy provides a versatile tool for investigating electron dynamics in gaseous, liquid, and solid samples on sub-femtosecond time scales. The extraction of information from spectrograms recorded with the attosecond streak camera remains a difficult challenge. Common algorithms are highly specialized and typically computationally heavy. In this work, we apply deep neural networks to map from streaking traces to near-infrared pulses as well as electron wavepackets and extensively benchmark our results on simulated data. Additionally, we illustrate domain-shift to real-world data. We also attempt to quantify the model predictive uncertainty. Our deep neural networks display competitive retrieval quality and superior tolerance against noisy data conditions, while reducing the computational time by orders of magnitude.

Accelerating in vivo fast spin echo high angular resolution diffusion imaging with an isotropic resolution in mice through compressed sensing.

PURPOSE: Echo planar imaging (EPI) is commonly used to acquire the many volumes needed for high angular resolution diffusion Imaging (HARDI), posing a higher risk for artifacts, such as distortion and deformation. An alternative to EPI is fast spin echo (FSE) imaging, which has fewer artifacts but is inherently slower. The aim is to accelerate FSE such that a HARDI data set can be acquired in a time comparable to EPI using compressed sensing. METHODS: Compressed sensing was applied in either q-space or simultaneously in k-space and q-space, by undersampling the k-space in the phase-encoding direction or retrospectively eliminating diffusion directions for different degrees of undersampling. To test the replicability of the acquisition and reconstruction, brain data were acquired from six mice, and a numerical phantom experiment was performed. All HARDI data were analyzed individually using constrained spherical deconvolution, and the apparent fiber density and complexity metric were evaluated, together with whole-brain tractography. RESULTS: The apparent fiber density and complexity metric showed relatively minor differences when only q-space undersampling was used, but deteriorate when k-space undersampling was applied. Likewise, the tract density weighted image showed good results when only q-space undersampling was applied using 15 directions or more, but information was lost when fewer volumes or k-space undersampling were used. CONCLUSION: It was found that acquiring 15 to 20 diffusion directions with a full k-space and reconstructed using compressed sensing could suffice for a replicable measurement of quantitative measures in mice, where areas near the sinuses and ear cavities are untainted by signal loss.

Field phenotyping of grapevine growth using dense stereo reconstruction.

BACKGROUND: The demand for high-throughput and objective phenotyping in plant research has been increasing during the last years due to large experimental sites. Sensor-based, non-invasive and automated processes are needed to overcome the phenotypic bottleneck, which limits data volumes on account of manual evaluations. A major challenge for sensor-based phenotyping in vineyards is the distinction between the grapevine in the foreground and the field in the background - this is especially the case for red-green-blue (RGB) images, where similar color distributions occur both in the foreground plant and in the field and background plants. However, RGB cameras are a suitable tool in the field because they provide high-resolution data at fast acquisition rates with robustness to outdoor illumination. RESULTS: This study presents a method to segment the phenotypic classes 'leaf', 'stem', 'grape' and 'background' in RGB images that were taken with a standard consumer camera in vineyards. Background subtraction is achieved by taking two images of each plant for depth reconstruction. The color information is furthermore used to distinguish the leaves from stem and grapes in the foreground. The presented approach allows for objective computation of phenotypic traits like 3D leaf surface areas and fruit-to-leaf ratios. The method has been successfully applied to objective assessment of growth habits of new breeding lines. To this end, leaf areas of two breeding lines were monitored and compared with traditional cultivars. A statistical analysis of the method shows a significant (p <0.001) determination coefficient R (2)= 0.93 and root-mean-square error of 3.0%. CONCLUSIONS: The presented approach allows for non-invasive, fast and objective assessment of plant growth. The main contributions of this study are 1) the robust segmentation of RGB images taken from a standard consumer camera directly in the field, 2) in particular, the robust background subtraction via reconstruction of dense depth maps, and 3) phenotypic applications to monitoring of plant growth and computation of fruit-to-leaf ratios in 3D. This advance provides a promising tool for high-throughput, automated image acquisition, e.g., for field robots.

Efficient Noninvasive FHB Estimation using RGB Images from a Novel Multiyear, Multirater Dataset.

Fusarium head blight (FHB) is one of the most prevalent wheat diseases, causing substantial yield losses and health risks. Efficient phenotyping of FHB is crucial for accelerating resistance breeding, but currently used methods are time-consuming and expensive. The present article suggests a noninvasive classification model for FHB severity estimation using red-green-blue (RGB) images, without requiring extensive preprocessing. The model accepts images taken from consumer-grade, low-cost RGB cameras and classifies the FHB severity into 6 ordinal levels. In addition, we introduce a novel dataset consisting of around 3,000 images from 3 different years (2020, 2021, and 2022) and 2 FHB severity assessments per image from independent raters. We used a pretrained EfficientNet (size b0), redesigned as a regression model. The results demonstrate that the interrater reliability (Cohen's kappa, κ) is substantially lower than the achieved individual network-to-rater results, e.g., 0.68 and 0.76 for the data captured in 2020, respectively. The model shows a generalization effect when trained with data from multiple years and tested on data from an independent year. Thus, using the images from 2020 and 2021 for training and 2022 for testing, we improved the F1w score by 0.14, the accuracy by 0.11, κ by 0.12, and reduced the root mean squared error by 0.5 compared to the best network trained only on a single year's data. The proposed lightweight model and methods could be deployed on mobile devices to automatically and objectively assess FHB severity with images from low-cost RGB cameras. The source code and the dataset are available at https://github.com/cvims/FHB_classification.

An image classification approach to analyze the suppression of plant immunity by the human pathogen Salmonella Typhimurium.

BACKGROUND: The enteric pathogen Salmonella is the causative agent of the majority of food-borne bacterial poisonings. Resent research revealed that colonization of plants by Salmonella is an active infection process. Salmonella changes the metabolism and adjust the plant host by suppressing the defense mechanisms. In this report we developed an automatic algorithm to quantify the symptoms caused by Salmonella infection on Arabidopsis. RESULTS: The algorithm is designed to attribute image pixels into one of the two classes: healthy and unhealthy. The task is solved in three steps. First, we perform segmentation to divide the image into foreground and background. In the second step, a support vector machine (SVM) is applied to predict the class of each pixel belonging to the foreground. And finally, we do refinement by a neighborhood-check in order to omit all falsely classified pixels from the second step. The developed algorithm was tested on infection with the non-pathogenic E. coli and the plant pathogen Pseudomonas syringae and used to study the interaction between plants and Salmonella wild type and T3SS mutants. We proved that T3SS mutants of Salmonella are unable to suppress the plant defenses. Results obtained through the automatic analyses were further verified on biochemical and transcriptome levels. CONCLUSION: This report presents an automatic pixel-based classification method for detecting "unhealthy" regions in leaf images. The proposed method was compared to existing method and showed a higher accuracy. We used this algorithm to study the impact of the human pathogenic bacterium Salmonella Typhimurium on plants immune system. The comparison between wild type bacteria and T3SS mutants showed similarity in the infection process in animals and in plants. Plant epidemiology is only one possible application of the proposed algorithm, it can be easily extended to other detection tasks, which also rely on color information, or even extended to other features.

Learn to Predict Sets Using Feed-Forward Neural Networks.

This paper addresses the task of set prediction using deep feed-forward neural networks. A set is a collection of elements which is invariant under permutation and the size of a set is not fixed in advance. Many real-world problems, such as image tagging and object detection, have outputs that are naturally expressed as sets of entities. This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors. We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks. In our formulation we define a likelihood for a set distribution represented by a) two discrete distributions defining the set cardinally and permutation variables, and b) a joint distribution over set elements with a fixed cardinality. Depending on the problem under consideration, we define different training models for set prediction using deep neural networks. We demonstrate the validity of our set formulations on relevant vision problems such as: 1) multi-label image classification where we outperform the other competing methods on the PASCAL VOC and MS COCO datasets, 2) object detection, for which our formulation outperforms popular state-of-the-art detectors, and 3) a complex CAPTCHA test, where we observe that, surprisingly, our set-based network acquired the ability of mimicking arithmetics without any rules being coded.

Photometric Depth Super-Resolution.

This study explores the use of photometric techniques (shape-from-shading and uncalibrated photometric stereo) for upsampling the low-resolution depth map from an RGB-D sensor to the higher resolution of the companion RGB image. A single-shot variational approach is first put forward, which is effective as long as the target's reflectance is piecewise-constant. It is then shown that this dependency upon a specific reflectance model can be relaxed by focusing on a specific class of objects (e.g., faces), and delegate reflectance estimation to a deep neural network. A multi-shot strategy based on randomly varying lighting conditions is eventually discussed. It requires no training or prior on the reflectance, yet this comes at the price of a dedicated acquisition setup. Both quantitative and qualitative evaluations illustrate the effectiveness of the proposed methods on synthetic and real-world scenarios.

A Region-Based Gauss-Newton Approach to Real-Time Monocular Multiple Object Tracking.

We propose an algorithm for real-time 6DOF pose tracking of rigid 3D objects using a monocular RGB camera. The key idea is to derive a region-based cost function using temporally consistent local color histograms. While such region-based cost functions are commonly optimized using first-order gradient descent techniques, we systematically derive a Gauss-Newton optimization scheme which gives rise to drastically faster convergence and highly accurate and robust tracking performance. We furthermore propose a novel complex dataset dedicated for the task of monocular object pose tracking and make it publicly available to the community. To our knowledge, it is the first to address the common and important scenario in which both the camera as well as the objects are moving simultaneously in cluttered scenes. In numerous experiments-including our own proposed dataset-we demonstrate that the proposed Gauss-Newton approach outperforms existing approaches, in particular in the presence of cluttered backgrounds, heterogeneous objects and partial occlusions.

A Noninvasive 3D Body Scanner and Software Tool towards Analysis of Scoliosis.

PURPOSE: Children with neurological disorders, such as cerebral palsy (CP), have a high risk of developing scoliosis during growth. The fast progression of scoliosis implies in several cases frequent clinical and X-ray examinations. We present an ionizing radiation-free, noncontacting method to estimate the trajectory of the vertebral column and to potentially facilitate medical diagnosis in cases where an X-ray examination is not indicated. METHODS: A body scanner and corresponding analysis software tools have been developed to get 3D surface scans of patient torsos and to analyze their spinal curvatures. The trajectory of the vertebral column has been deduced from the body contours at different transverse sectional planes along the vertical torso axis. In order to verify the present methods, we have analyzed twenty-five torso contours, extracted from computer tomography (CT) images of patients who had a CT scan for other medical reasons, but incidentally also showed a scoliosis. The software tools therefore process data from the body scanner as well as X-ray or CT images. RESULTS: The methods presented show good results in the estimations of the lateral deviation of the spine for mild and moderate scoliosis. The partial mismatch for severe cases is associated with a less accurate estimation of the rotation of the vertebrae around the vertical body axis in these cases. In addition, distinct torso contour shapes, in the transverse sections, have been characterized according to the severity of the scoliosis. CONCLUSION: The hardware and software tools are a first step towards an ionizing radiation-free analysis of progression of scoliosis. However, further improvements of the analysis methods and tests on a larger number of data sets with diverse types of scoliosis are necessary, before its introduction into clinical application as a supplementary tool to conventional examinations.

Efficient nonlocal means for denoising of textural patterns.

This paper contributes two novel techniques in the context of image restoration by nonlocal filtering. First, we introduce an efficient implementation of the nonlocal means filter based on arranging the data in a cluster tree. The structuring of data allows for a fast and accurate preselection of similar patches. In contrast to previous approaches, the preselection is based on the same distance measure as used by the filter itself. It allows for large speedups, especially when the search for similar patches covers the whole image domain, i.e., when the filter is truly nonlocal. However, also in the windowed version of the filter, the cluster tree approach compares favorably to previous techniques in respect of quality versus computational cost. Second, we suggest an iterative version of the filter that is derived from a variational principle and is designed to yield nontrivial steady states. It reveals to be particularly useful in order to restore regular, textured patterns.

Direct Sparse Odometry.

Direct Sparse Odometry (DSO) is a visual odometry method based on a novel, highly accurate sparse and direct structure and motion formulation. It combines a fully direct probabilistic model (minimizing a photometric error) with consistent, joint optimization of all model parameters, including geometry-represented as inverse depth in a reference frame-and camera motion. This is achieved in real time by omitting the smoothness prior used in other direct methods and instead sampling pixels evenly throughout the images. Since our method does not depend on keypoint detectors or descriptors, it can naturally sample pixels from across all image regions that have intensity gradient, including edges or smooth intensity variations on essentially featureless walls. The proposed model integrates a full photometric calibration, accounting for exposure time, lens vignetting, and non-linear response functions. We thoroughly evaluate our method on three different datasets comprising several hours of video. The experiments show that the presented approach significantly outperforms state-of-the-art direct and indirect methods in a variety of real-world settings, both in terms of tracking accuracy and robustness.

Radiomics in radiooncology - Challenging the medical physicist.

PURPOSE: Noticing the fast growing translation of artificial intelligence (AI) technologies to medical image analysis this paper emphasizes the future role of the medical physicist in this evolving field. Specific challenges are addressed when implementing big data concepts with high-throughput image data processing like radiomics and machine learning in a radiooncology environment to support clinical decisions. METHODS: Based on the experience of our interdisciplinary radiomics working group, techniques for processing minable data, extracting radiomics features and associating this information with clinical, physical and biological data for the development of prediction models are described. A special emphasis was placed on the potential clinical significance of such an approach. RESULTS: Clinical studies demonstrate the role of radiomics analysis as an additional independent source of information with the potential to influence the radiooncology practice, i.e. to predict patient prognosis, treatment response and underlying genetic changes. Extending the radiomics approach to integrate imaging, clinical, genetic and dosimetric data ('panomics') challenges the medical physicist as member of the radiooncology team. CONCLUSIONS: The new field of big data processing in radiooncology offers opportunities to support clinical decisions, to improve predicting treatment outcome and to stimulate fundamental research on radiation response both of tumor and normal tissue. The integration of physical data (e.g. treatment planning, dosimetric, image guidance data) demands an involvement of the medical physicist in the radiomics approach of radiooncology. To cope with this challenge national and international organizations for medical physics should organize more training opportunities in artificial intelligence technologies in radiooncology.

Genetic defects in ß-spectrin and tau sensitize C. elegans axons to movement-induced damage via torque-tension coupling.

Our bodies are in constant motion and so are the neurons that invade each tissue. Motion-induced neuron deformation and damage are associated with several neurodegenerative conditions. Here, we investigated the question of how the neuronal cytoskeleton protects axons and dendrites from mechanical stress, exploiting mutations in UNC-70 ß-spectrin, PTL-1 tau/MAP2-like and MEC-7 ß-tubulin proteins in Caenorhabditis elegans. We found that mechanical stress induces supercoils and plectonemes in the sensory axons of spectrin and tau double mutants. Biophysical measurements, super-resolution, and electron microscopy, as well as numerical simulations of neurons as discrete, elastic rods provide evidence that a balance of torque, tension, and elasticity stabilizes neurons against mechanical deformation. We conclude that the spectrin and microtubule cytoskeletons work in combination to protect axons and dendrites from mechanical stress and propose that defects in ß-spectrin and tau may sensitize neurons to damage.

Dynamical statistical shape priors for level set-based tracking.

In recent years, researchers have proposed introducing statistical shape knowledge into level set-based segmentation methods in order to cope with insufficient low-level information. While these priors were shown to drastically improve the segmentation of familiar objects, so far the focus has been on statistical shape priors which are static in time. Yet, in the context of tracking deformable objects, it is clear that certain silhouettes (such as those of a walking person) may become more or less likely over time. In this paper, we tackle the challenge of learning dynamical statistical models for implicitly represented shapes. We show how these can be integrated as dynamical shape priors in a Bayesian framework for level set-based image sequence segmentation. We assess the effect of such shape priors "with memory" on the tracking of familiar deformable objects in the presence of noise and occlusion. We show comparisons between dynamical and static shape priors, between models of pure deformation and joint models of deformation and transformation, and we quantitatively evaluate the segmentation accuracy as a function of the noise level and of the camera frame rate. Our experiments demonstrate that level set-based segmentation and tracking can be strongly improved by exploiting the temporal correlations among consecutive silhouettes which characterize deforming shapes.

Integral invariants for shape matching.

For shapes represented as closed planar contours, we introduce a class of functionals which are invariant with respect to the Euclidean group and which are obtained by performing integral operations. While such integral invariants enjoy some of the desirable properties of their differential counterparts, such as locality of computation (which allows matching under occlusions) and uniqueness of representation (asymptotically), they do not exhibit the noise sensitivity associated with differential quantities and, therefore, do not require presmoothing of the input shape. Our formulation allows the analysis of shapes at multiple scales. Based on integral invariants, we define a notion of distance between shapes. The proposed distance measure can be computed efficiently and allows warping the shape boundaries onto each other; its computation results in optimal point correspondence as an intermediate step. Numerical results on shape matching demonstrate that this framework can match shapes despite the deformation of subparts, missing parts and noise. As a quantitative analysis, we report matching scores for shape retrieval from a database.

q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI Scans.

Numerous scientific fields rely on elaborate but partly suboptimal data processing pipelines. An example is diffusion magnetic resonance imaging (diffusion MRI), a non-invasive microstructure assessment method with a prominent application in neuroimaging. Advanced diffusion models providing accurate microstructural characterization so far have required long acquisition times and thus have been inapplicable for children and adults who are uncooperative, uncomfortable, or unwell. We show that the long scan time requirements are mainly due to disadvantages of classical data processing. We demonstrate how deep learning, a group of algorithms based on recent advances in the field of artificial neural networks, can be applied to reduce diffusion MRI data processing to a single optimized step. This modification allows obtaining scalar measures from advanced models at twelve-fold reduced scan time and detecting abnormalities without using diffusion models. We set a new state of the art by estimating diffusion kurtosis measures from only 12 data points and neurite orientation dispersion and density measures from only 8 data points. This allows unprecedentedly fast and robust protocols facilitating clinical routine and demonstrates how classical data processing can be streamlined by means of deep learning.

Variational Depth From Focus Reconstruction.

This paper deals with the problem of reconstructing a depth map from a sequence of differently focused images, also known as depth from focus (DFF) or shape from focus. We propose to state the DFF problem as a variational problem, including a smooth but nonconvex data fidelity term and a convex nonsmooth regularization, which makes the method robust to noise and leads to more realistic depth maps. In addition, we propose to solve the nonconvex minimization problem with a linearized alternating directions method of multipliers, allowing to minimize the energy very efficiently. A numerical comparison to classical methods on simulated as well as on real data is presented.

Spatially varying color distributions for interactive multilabel segmentation.

We propose a method for interactive multilabel segmentation which explicitly takes into account the spatial variation of color distributions. To this end, we estimate a joint distribution over color and spatial location using a generalized Parzen density estimator applied to each user scribble. In this way, we obtain a likelihood for observing certain color values at a spatial coordinate. This likelihood is then incorporated in a Bayesian MAP estimation approach to multiregion segmentation which in turn is optimized using recently developed convex relaxation techniques. These guarantee global optimality for the two-region case (foreground/background) and solutions of bounded optimality for the multiregion case. We show results on the GrabCut benchmark, the recently published Graz benchmark, and on the Berkeley segmentation database which exceed previous approaches such as GrabCut, the Random Walker, Santner's approach, TV-Seg, and interactive graph cuts in accuracy. Our results demonstrate that taking into account the spatial variation of color models leads to drastic improvements for interactive image segmentation.

A coding-cost framework for super-resolution motion layer decomposition.

We consider the problem of decomposing a video sequence into a superposition of (a given number of) moving layers. For this problem, we propose an energy minimization approach based on the coding cost. Our contributions affect both the model (what is minimized) and the algorithmic side (how it is minimized). The novelty of the coding-cost model is the inclusion of a refined model of the image formation process, known as super resolution. This accounts for camera blur and area averaging arising in a physically plausible image formation process. It allows us to extract sharp high-resolution layers from the video sequence. The algorithmic framework is based on an alternating minimization scheme and includes the following innovations. 1) A video labeling, we optimize the layer domains. This allows to regularize the shapes of the layers and a very elegant handling of occlusions. 2) We present an efficient parallel algorithm for extracting super-resolved layers based on TV filtering.

Fast joint estimation of silhouettes and dense 3D geometry from multiple images.

We propose a probabilistic formulation of joint silhouette extraction and 3D reconstruction given a series of calibrated 2D images. Instead of segmenting each image separately in order to construct a 3D surface consistent with the estimated silhouettes, we compute the most probable 3D shape that gives rise to the observed color information. The probabilistic framework, based on Bayesian inference, enables robust 3D reconstruction by optimally taking into account the contribution of all views. We solve the arising maximum a posteriori shape inference in a globally optimal manner by convex relaxation techniques in a spatially continuous representation. For an interactively provided user input in the form of scribbles specifying foreground and background regions, we build corresponding color distributions as multivariate Gaussians and find a volume occupancy that best fits to this data in a variational sense. Compared to classical methods for silhouette-based multiview reconstruction, the proposed approach does not depend on initialization and enjoys significant resilience to violations of the model assumptions due to background clutter, specular reflections, and camera sensor perturbations. In experiments on several real-world data sets, we show that exploiting a silhouette coherency criterion in a multiview setting allows for dramatic improvements of silhouette quality over independent 2D segmentations without any significant increase of computational efforts. This results in more accurate visual hull estimation, needed by a multitude of image-based modeling approaches. We made use of recent advances in parallel computing with a GPU implementation of the proposed method generating reconstructions on volume grids of more than 20 million voxels in up to 4.41 seconds.