Machine learning-based estimation of spatial gene expression pattern during ESC-derived retinal organoid development.

Organoids, which can reproduce the complex tissue structures found in embryos, are revolutionizing basic research and regenerative medicine. In order to use organoids for research and medicine, it is necessary to assess the composition and arrangement of cell types within the organoid, i.e., spatial gene expression. However, current methods are invasive and require gene editing and immunostaining. In this study, we developed a non-invasive estimation method of spatial gene expression patterns using machine learning. A deep learning model with an encoder-decoder architecture was trained on paired datasets of phase-contrast and fluorescence images, and was applied to a retinal organoid derived from mouse embryonic stem cells, focusing on the master gene Rax (also called Rx), crucial for eye field development. This method successfully estimated spatially plausible fluorescent patterns with appropriate intensities, enabling the non-invasive, quantitative estimation of spatial gene expression patterns within each tissue. Thus, this method could lead to new avenues for evaluating spatial gene expression patterns across a wide range of biology and medicine fields.

Unsupervised learning with a physics-based autoencoder for estimating the thickness and mixing ratio of pigments.

Layered surface objects represented by decorated tomb murals and watercolors are in danger of deterioration and damage. To address these dangers, it is necessary to analyze the pigments' thickness and mixing ratio and record the current status. This paper proposes an unsupervised autoencoder model for thickness and mixing ratio estimation. The input of our autoencoder is spectral data of layered surface objects. Our autoencoder is unique, to our knowledge, in that the decoder part uses a physical model, the Kubelka-Munk model. Since we use the Kubelka-Munk model for the decoder, latent variables in the middle layer can be interpretable as the pigment thickness and mixing ratio. We conducted a quantitative evaluation using synthetic data and confirmed that our autoencoder provides a highly accurate estimation. We measured an object with layered surface pigments for qualitative evaluation and confirmed that our method is valid in an actual environment. We also present the superiority of our unsupervised autoencoder over supervised learning.

Descattering for transmissive inspection in production line using slanted linear image sensors.

We propose a descattering method that can be easily applied to food production lines. The system consists of several sets of linear image sensors and linear light sources slanted at different angles. The images captured by these sensors are partially clear along the direction perpendicular to the sensors. We computationally integrate these images on the frequency domain into a single clear image. The effectiveness of the proposed method is assessed by simulation and real-world experiments. The results show that our method recovers clear images. We demonstrate the applicability of the proposed method to a real production line by a prototype system.

Super-resolution for a dispersive spectrometer using a tilted area sensor and spectrally varying blur kernel interpolation.

The grating, lens, and linear sensor determine a spectrometer's wavelength resolution and measurement range. While conventional methods have tried to improve the optical design to obtain a better resolution, they have a limitation caused by the physical property. To improve the resolution, we introduce a super-resolution method from the computer vision field. We propose tilting an area sensor to realize accurate subpixel shifting and recover a high-resolution spectrum using interpolated spectrally varying kernels. We experimentally validate that the proposed method achieved a high spectral resolution of 0.141nm in 400-800nm by just tilting the sensor in the spectrometer.

Time-of-flight imaging in fog using multiple time-gated exposures.

We propose a time-of-flight measurement algorithm for depth and intensity that is robust to fog. The key idea of the algorithm is to compensate for the scattering effects of fog by using multiple time-gating and assigning one time-gated exposure for scattering property estimation. Once the property is estimated, the depth and intensity can be reconstructed from the rest of the exposures via a physics-based model. Several experiments with artificial fog show that our method can measure depth and intensity irrespective of the traits of the fog. We also confirm the effectiveness of our method in real fog through an outdoor experiment.

Programmable Non-Epipolar Indirect Light Transport: Capture and Analysis.

The decomposition of light transport into direct and global components, diffuse and specular interreflections, and subsurface scattering allows for new visualizations of light in everyday scenes. In particular, indirect light contains a myriad of information about the complex appearance of materials useful for computer vision and inverse rendering applications. In this paper, we present a new imaging technique that captures and analyzes components of indirect light via light transport using a synchronized projector-camera system. The rectified system illuminates the scene with epipolar planes corresponding to projector rows, and we vary two key parameters to capture plane-to-ray light transport between projector row and camera pixel: (1) the offset between projector row and camera row in the rolling shutter (implemented as synchronization delay), and (2) the exposure of the camera row. We describe how this synchronized rolling shutter performs illumination multiplexing, and develop a nonlinear optimization algorithm to demultiplex the resulting 3D light transport operator. Using our system, we are able to capture live short and long-range non-epipolar indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties. In particular, we show the utility of indirect imaging for capturing and analyzing the hidden structure of veins in human skin.

Time-Resolved Far Infrared Light Transport Decomposition for Thermal Photometric Stereo.

We present a novel time-resolved light transport decomposition method using thermal imaging. Because the speed of heat propagation is much slower than the speed of light propagation, the transient transport of far infrared light can be observed at a video frame rate. A key observation is that the thermal image looks similar to the visible light image in an appropriately controlled environment. This implies that conventional computer vision techniques can be straightforwardly applied to the thermal image. We show that the diffuse component in the thermal image can be separated, and therefore, the surface normals of objects can be estimated by the Lambertian photometric stereo. The effectiveness of our method is evaluated by conducting real-world experiments, and its applicability to black body, transparent, and translucent objects is shown.

Three-dimensional morphogenesis of the omental bursa from four recesses in staged human embryos.

The omental bursa (OB) is a complex upper abdominal structure in adults. Its morphological complexity stems from embryonic development. Approximately 200 years ago, the first theory regarding OB development was reported, describing that the OB developed from changes in the position of the stomach and its dorsal mesentery. Thereafter, the second theory reported that the OB originated from three recesses: the right pneumato-enteric recess (rPER), hepato-enteric recess (HER), and pancreatico-enteric recess (PaER). However, the first theory, focusing on the rotation of the stomach, is still described in certain modern embryology textbooks. These two coexisting embryological theories deter the understanding of the anatomical complexity of the OB. This study aimed to unify these two theories into realistic illustrations. Approximately 10 samples per stage among Carnegie stage (CS) 13 and CS21 were microscopically observed and histological serial sections of the representative samples were aligned using the new automatic alignment method. The aligned images were segmented computationally and reconstructed into 3D models. The rPER and the HER encompassed the right half circumference of the esophagus and the stomach at CS13 and CS14, the PaER spread dorsal to the stomach and formed a discoid shape at CS15 and CS16, the infracardiac bursa (ICB) was separated by the diaphragm at CS17 and CS18, and the fourth recess, which we called the greater omental recess (GOR), extended caudally from the PaER among CS19 and CS21. The present results indicate that the fourth recess is also the origin of the OB. These two theories over 200 years can be generally unified into one embryological description indicating a new recess as the origin of the OB.

Depth from phasor distortions in fog.

This paper presents a time-of-flight (ToF) measurement method for use in foggy weather. The depth measured by a ToF camera is greatly distorted in fog because the light scattered in the fog reaches the camera much faster than the target reflection. We reveal that the multi-frequency measurements contain a cue whether two arbitrary pixels have the same depth. After clustering the same depth pixels using this cue, the original depth can be recovered for each cluster by line fitting in the Cartesian coordinate frame. The effectiveness of this method is evaluated numerically via real-world and road-scale experiments.

Material Classification from Time-of-Flight Distortions.

This paper presents a material classification method using an off-the-shelf Time-of-Flight (ToF) camera. The proposed method is built upon a key observation that the depth measurement by a ToF camera is distorted for objects with certain materials, especially with translucent materials. We show that this distortion is due to the variation of time domain impulse responses across materials and also due to the measurement mechanism of the ToF cameras. Specifically, we reveal that the amount of distortion varies according to the modulation frequency of the ToF camera, the object material, and the distance between the camera and object. Our method uses the depth distortion of ToF measurements as a feature for classification and achieves material classification of a scene. Effectiveness of the proposed method is demonstrated by numerical evaluations and real-world experiments, showing its capability of material classification, even for visually indistinguishable objects.

Variational Bayesian Approach to Multiframe Image Restoration.

Image restoration is a fundamental problem in the field of image processing. The key objective of image restoration is to recover clean images from images degraded by noise and blur. Recently, a family of new statistical techniques called variational Bayes (VB) has been introduced to image restoration, which enables us to automatically tune parameters that control restoration. While information from one image is often insufficient for high-quality restoration, however, current state-of-the-art methods of image restoration via VB approaches use only a single-degraded image to recover a clean image. In this paper, we propose a novel method of multiframe image restoration via a VB approach, which can achieve higher image quality while tuning parameters automatically. Given multiple degraded images, this method jointly estimates a clean image and other parameters, including an image warping parameter introduced for the use of multiple images, through Bayesian inference that we enable by making full use of VB techniques. Through various experiments, we demonstrate the effectiveness of our multiframe method by comparing it with single-frame one, and also show the advantages of our VB approach over non-VB approaches.