Calibration-free structured-light-based 3D scanning system in laparoscope for robotic surgery.

Accurate 3D shape measurement is crucial for surgical support and alignment in robotic surgery systems. Stereo cameras in laparoscopes offer a potential solution; however, their accuracy in stereo image matching diminishes when the target image has few textures. Although stereo matching with deep learning has gained significant attention, supervised learning requires a large dataset of images with depth annotations, which are scarce for laparoscopes. Thus, there is a strong demand to explore alternative methods for depth reconstruction or annotation for laparoscopes. Active stereo techniques are a promising approach for achieving 3D reconstruction without textures. In this study, a 3D shape reconstruction method is proposed using an ultra-small patterned projector attached to a laparoscopic arm to address these issues. The pattern projector emits a structured light with a grid-like pattern that features node-wise modulation for positional encoding. To scan the target object, multiple images are taken while the projector is in motion, and the relative poses of the projector and a camera are auto-calibrated using a differential rendering technique. In the experiment, the proposed method is evaluated by performing 3D reconstruction using images obtained from a surgical robot and comparing the results with a ground-truth shape obtained from X-ray CT.

Scale-preserving shape reconstruction from monocular endoscope image sequences by supervised depth learning.

Reconstructing 3D shapes from images are becoming popular, but such methods usually estimate relative depth maps with ambiguous scales. A method for reconstructing a scale-preserving 3D shape from monocular endoscope image sequences through training an absolute depth prediction network is proposed. First, a dataset of synchronized sequences of RGB images and depth maps is created using an endoscope simulator. Then, a supervised depth prediction network is trained that estimates a depth map from a RGB image minimizing the loss compared to the ground-truth depth map. The predicted depth map sequence is aligned to reconstruct a 3D shape. Finally, the proposed method is applied to a real endoscope image sequence.

A Generative Model to Embed Human Expressivity into Robot Motions.

This paper presents a model for generating expressive robot motions based on human expressive movements. The proposed data-driven approach combines variational autoencoders and a generative adversarial network framework to extract the essential features of human expressive motion and generate expressive robot motion accordingly. The primary objective was to transfer the underlying expressive features from human to robot motion. The input to the model consists of the robot task defined by the robot's linear velocities and angular velocities and the expressive data defined by the movement of a human body part, represented by the acceleration and angular velocity. The experimental results show that the model can effectively recognize and transfer expressive cues to the robot, producing new movements that incorporate the expressive qualities derived from the human input. Furthermore, the generated motions exhibited variability with different human inputs, highlighting the ability of the model to produce diverse outputs.

Single and multi-frame auto-calibration for 3D endoscopy with differential rendering.

The use of 3D measurement in endoscopic images offers practicality in cancer diagnosis, computer-assisted interventions, and making annotations for machine learning training data. An effective approach is the implementation of an active stereo system, using a micro-sized pattern projector and an endoscope camera, which has been intensively developed. One open problem for such a system is the necessity of strict and complex calibration of the projector-camera system to precisely recover the shapes. Moreover, since the head of an endoscope should have enough elasticity to avoid harming target objects, the positions of the pattern projector cannot be tightly fixed to the head, resulting in limited accuracy. A straightforward approach to the problem is applying auto-calibration. However, it requires special markers in the pattern or a highly accurate initial position for stable calibration, which is impractical for real operation. In the paper, we propose a novel auto-calibration method based on differential rendering techniques, which are recently proposed and drawing wide attention. To apply the method to an endoscopic system, where a diffractive optical element (DOE) is used, we propose a technique to simultaneously estimate the focal length of the DOE as well as the extrinsic parameters between a projector and a camera. We also propose a multi-frame optimization algorithm to jointly optimize the intrinsic and extrinsic parameters, relative pose between frames, and the entire shape.Clinical relevance- One-shot endoscopic measurement of depth information is a practical solution for cancer diagnosis, computer-assisted interventions, and making annotations for machine learning training data.

Wavy Graphene-Like Network Forming during Pyrolysis of Polyacrylonitrile into Carbon Fiber.

Carbon fiber (CF) obtained by pyrolysis of polyacrylonitrile (PAN-CF) surpasses metals in properties suitable for diverse applications such as aircraft manufacture and power turbine blades. PAN-CF obtained by pyrolysis at 1200-1400 °C shows a remarkably high tensile strength of 7 GPa, much higher than pitch-based CF (pb-CF) consisting of piles of pure graphene networks. However, little information has been available on the atomistic structure of PAN-CF and on how it forms during pyrolysis. We pyrolyzed an acrylonitrile 9-mer in a carbon nanotube, monitored the course of the reaction using atomic-resolution electron microscopy and Raman spectroscopy, and found that this oligomer forms a thermally reactive wavy graphene-like network (WGN) at 1200-1400 °C during slow graphitization taking place between 900 and 1800 °C. Ptychographic microscopic analysis indicated that such material consists of 5-, 6-, and larger-membered rings; hence, it is not flat but wavy. The experimental data suggest that, during PAN-CF manufacturing, many layers of WGN hierarchically pile up to form a chemically and physically interdigitated noncrystalline phase that resists fracture and increases the tensile strength─the properties expected for high-entropy materials. pb-CF using nearly pure carbon starting material, on the other hand, forms a crystalline graphene network and is brittle.

Direct observation of Cu in high-silica chabazite zeolite by electron ptychography using Wigner distribution deconvolution.

Direct observation of Cu in Cu-chabazite (CHA) zeolite has been achieved by electron ptychography using the Wigner distribution deconvolution. The imaging properties of ptychographically reconstructed images were evaluated by comparing the intensities of six-membered-ring columns of the zeolite with and without Cu using simulated ptychography images. It was concluded that although false contrast may appear at Cu-free columns for some acquisition conditions, ptychography can discriminate columns with and without Cu. Experimental observation of CHA with and without Cu was performed. Images obtained from the Cu-containing sample showed contrast at the six-membered-rings, while no contrast was observed for the Cu-free sample. The results show that ptychography is a promising technique for visualizing the atomic structures of beam-sensitive materials.

3D endoscope system with AR display superimposing dense and wide-angle-of-view 3D points obtained by using micro pattern projector.

In recent years, augmented reality (AR) technologies have been widespread for supporting various kinds of tasks, by superimposing useful information on the users' view of the real environments. In endoscopic diagnosis, AR systems can be helpful as an aid in presenting information to endoscopists who have their hands full. In this paper, we propose a system that can superimpose shapes, which are reconstructed from an endoscope image, onto the field of view. The feature of the proposed system is that it reconstructs 3D shapes from the images captured by the endoscope and superimposes them onto the real views. As a result, the superimposed view allows the doctor to keep operating the endoscope while observing the patient's internal body with additional information. The proposed system is composed of the reconstruction module and the display module. The reconstruction module is for acquiring 3D shapes based on an active stereo method. In particular, we propose a novel projection pattern that can reconstruct wide areas of the endoscopic view. The display module shows the 3D shape obtained by the reconstructed module, superimposing on the field of view. In the experiments, we show that it is possible to perform a wide range of dense 3D reconstructions using the new projection patterns. In addition, we confirmed the usefulness of the AR system by interviewing medical doctors.

Estimating Muscle Activity from the Deformation of a Sequential 3D Point Cloud.

Estimation of muscle activity is very important as it can be a cue to assess a person's movements and intentions. If muscle activity states can be obtained through non-contact measurement, through visual measurement systems, for example, muscle activity will provide data support and help for various study fields. In the present paper, we propose a method to predict human muscle activity from skin surface strain. This requires us to obtain a 3D reconstruction model with a high relative accuracy. The problem is that reconstruction errors due to noise on raw data generated in a visual measurement system are inevitable. In particular, the independent noise between each frame on the time series makes it difficult to accurately track the motion. In order to obtain more precise information about the human skin surface, we propose a method that introduces a temporal constraint in the non-rigid registration process. We can achieve more accurate tracking of shape and motion by constraining the point cloud motion over the time series. Using surface strain as input, we build a multilayer perceptron artificial neural network for inferring muscle activity. In the present paper, we investigate simple lower limb movements to train the network. As a result, we successfully achieve the estimation of muscle activity via surface strain.

Active Stereo Method for 3D Endoscopes using Deep-layer GCN and Graph Representation with Proximity Information.

Techniques for 3D endoscopic systems have been widely studied for various reasons. Among them, active stereo based systems, in which structured-light patterns are projected to surfaces and endoscopic images of the pattern are analyzed to produce 3D depth images, are promising, because of robustness and simple system configurations. For those systems, finding correspondences between a projected pattern and an original pattern is an open problem. Recently, correspondence estimation by graph neural networks (GCN) using graph-based representation of the patterns were proposed for 3D endoscopic systems. One severe problem of the approach is that the graph matching by GCN is largely affected by the stability of the graph construction process using the detected patterns of a captured image. If the detected pattern is fragmented into small pieces, graph matching may fail and 3D shapes cannot be retrieved. In this paper, we propose a solution for those problems by applying deep-layered GCN and extended graph representations of the patterns, where proximity information is added. Experiments show that the proposed method outperformed the previous method in accuracies for correspondence matching for 3D reconstruction.

Exploiting the full potential of the advanced two-hexapole corrector for STEM exemplified at 60kV.

Ultimate resolution in scanning transmission electron microscopy (STEM) with state-of-the-art aberration correctors requires careful tuning of the experimental parameters. The optimum aperture semi-angle depends on the chosen high tension, the chromatic aberration and the energy width of the source as well as on potentially limiting intrinsic residual aberrations. In this paper we derive simple expressions and criteria for choosing the aperture semi-angle and for counterbalancing the intrinsic sixth-order three-lobe aberration of two-hexapole aberration correctors by means of the fourth-order three-lobe aberration. It is noteworthy that for such an optimally adjusted electron probe the so-called flat area of the Ronchigram is explicitly not maximized. The above considerations are validated by experiments with a CEOS ASCOR in a C-FEG-equipped JEOL NEOARM operated at 60 kV. Sub-Angstrom resolution is demonstrated for a Si[112] single crystal as well as for a single-layered MoS2 crystalline film. Lattice reflections of 73 pm for silicon and 93 pm for molybdenum disulfide are visible in the Fourier transform of the images, respectively. Moreover, single sulfur vacancies can be clearly identified in the MoS2.

Fully Auto-calibrated Active-stereo-based 3D Endoscopic System using Correspondence Estimation with Graph Convolutional Network.

We have developed a series of 3D endoscopic systems where a micro-sized pattern projector is inserted through the instrument channel of the endoscope and shapes are reconstructed by a structured light technique using captured images of the endoscopic camera. One problem of the previous works is that the accuracy of shape reconstruction is low, because the projector cannot be fixed to the endoscope, and thus, the pose of the pattern projector w.r.t. the camera cannot be pre-calibrated. In this paper, we propose a method to auto-calibrate the pose of the projector without using any special devices nor manual process. Since the technique is one-shot, multiple shapes can be reconstructed from an image sequence and a large 3D scene can be recovered by merging them. Experiments are conducted using the real system.

Electron ptychographic phase imaging of light elements in crystalline materials using Wigner distribution deconvolution.

Recent development in fast pixelated detector technology has allowed a two dimensional diffraction pattern to be recorded at every probe position of a two dimensional raster scan in a scanning transmission electron microscope (STEM), forming an information-rich four dimensional (4D) dataset. Electron ptychography has been shown to enable efficient coherent phase imaging of weakly scattering objects from a 4D dataset recorded using a focused electron probe, which is optimised for simultaneous incoherent Z-contrast imaging and spectroscopy in STEM. Therefore coherent phase contrast and incoherent Z-contrast imaging modes can be efficiently combined to provide a good sensitivity of both light and heavy elements at atomic resolution. In this work, we explore the application of electron ptychography for atomic resolution imaging of strongly scattering crystalline specimens, and present experiments on imaging crystalline specimens including samples containing defects, under dynamical channelling conditions using an aberration corrected microscope. A ptychographic reconstruction method called Wigner distribution deconvolution (WDD) was implemented. Experimental results and simulation results suggest that ptychography provides a readily interpretable phase image and great sensitivity for imaging light elements at atomic resolution in relatively thin crystalline materials.

3D endoscope system using DOE projector.

For effective in situ endoscopic diagnosis and treatment, size measurement and shape characterization of lesions, such as tumors, is important. For this purpose, in the past we have developed a range of 3D endoscopic systems based on active stereo to measure the shape and size of living tissues. In those works, the main shortcoming was that the target area could only be reconstructed at a specific distance from the scope because of off-focus blurring effects and aberrations in the periphery of the field of view. These issues were compounded by the degree of reconstruction instability due to the strong subsurface scattering common in internal tissue. In this paper, we tackle these shortcomings by developing a new micro pattern laser projector to be inserted in the scope tool channel. The new projector uses a Diffractive Optical Element (DOE) instead of a single lens, which solves the off-focus blur. We also propose a new line-based grid pattern with gap coding to counter the subsurface scattering effect. In our experiments on ex vivo human tumor samples, we show that the tissue shapes were successfully reconstructed regardless of depth variance and strong subsurface scattering effects.

2-DOF auto-calibration for a 3D endoscope system based on active stereo.

For endoscopic medical treatment, measuring the size and shape of lesions, such as tumors, is important. We are developing a 3D endoscope system to measure the shape and size of living tissues based on active stereo. In previous works, our group attached a pattern projector outside the endoscope head. Since this increased the diameter of the endoscope, the burden and the risks of the patients would increase. In this paper, we set the pattern projector inside the instrument channel of the endoscope instead of mounting it outside, so that it can be deployed whenever required. This does not increase the size of the endoscope and facilitates the measuring process. However, since the projector is not physically fixed to the endoscope anymore prior to the operation, we propose an "auto-calibration" technique where extrinsic parameters are calibrated intra-operatively from a point marker on the projector observed simultaneously on the target surface. In the experiment, we show that the external parameters were successfully calibrated to obtain 3D reconstructions properly with the overall systems. The accuracy of the auto-calibration was validated by confirming that the epipolar constraints were kept, and a 3D reconstruction of a human tissue was demonstrated.

Dense 3D Reconstruction from High Frame-Rate Video Using a Static Grid Pattern.

Dense 3D reconstruction of fast moving objects could contribute to various applications such as body structure analysis, accident avoidance, and so on. In this paper, we propose a technique based on a one-shot scanning method, which reconstructs 3D shapes for each frame of a high frame-rate video capturing the scenes projected by a static pattern. To avoid instability of image processing, we restrict the number of colors used in the pattern to less than two. The proposed technique comprises (1) an efficient algorithm to eliminate ambiguity of projected parallel-line patterns by using intersection points, (2) a batch reconstruction algorithm of multiple frames by using spatio-temporal constraints, and (3) an efficient detection method of color-encoded grid pattern based on de Bruijn sequence. In the experiments, the line detection algorithm worked effectively and the dense reconstruction algorithm produces accurate and robust results. We also show the improved results by using temporal constraints. Finally, the dense reconstructions of fast moving objects in a high frame-rate video are presented.

Calibration of a 3D endoscopic system based on active stereo method for shape measurement of biological tissues and specimen.

For endoscopic medical treatment, measuring the size and shape of the lesion, such as a tumor, is important for the improvement of diagnostic accuracy. We are developing a system to measure the shapes and sizes of living tissue by active stereo method using a normal endoscope on which a micro pattern projector is attached. In order to perform 3D reconstruction, estimating the intrinsic and extrinsic parameters of the endoscopic camera and the pattern projector is required. Particularly, calibration of the pattern projector is difficult. In this paper, we propose a simultaneous estimation method of both intrinsic and extrinsic parameters of the pattern projector. This simplifies the calibration procedure required in practical scenes. Furthermore, we have developed an efficient user interface to intuitively operate the calibration and reconstruction procedures. Using the developed system, we measured the shape of an internal tissue of the soft palate of a human and a biological specimen.

Noncontact measurement of cardiac beat by using active stereo with waved-grid pattern projection.

We propose a method to observe cardiac beat from 3D shape information of body surface by using the active stereo with waved-grid pattern projection, and report preliminary experiments to evaluate validities of the proposed method. By comparing results of our method with those of electrocardiogram (ECG), we confirmed sufficient correspondences between peak intervals of depth changes between contiguous frames measured by the active stereo and R-R intervals measured by ECG. We proposed the visualization of the spatial distribution of depth change plotted on the 3D shape of chest surface. We confirm that the spatial phase difference, which is caused by heart pump ability, appears in the 3-D shape change of chest surface.

Proposal on 3-D endoscope by using grid-based active stereo.

In this paper, we propose a novel 3-D endoscope system by using grid-based active stereo. In the proposed system, projection of a waved-grid pattern that consists of vertical and horizontal sinusoidal lines realizes accurate shape acquisition in sub-pixel accuracy. We develop a small pattern projector implementable to a head of a ready-made endoscope, and examine 3-D shape reconstruction by actual equipment. As the result of the measurement for a known-shaped object, which is a hexagonal cylinder, the error of length measurement is below 0.9% and the error of angle measurement is below 2.2%. We make a measurement of animal organ meat, and confirm that the system can reconstruct a 3-D shape of the organ surface.

A new type of molybdenum oxide crystal encapsulated inside a single-walled carbon nanotube.

The crystal structure of a new type of molybdenum oxide crystal encapsulated in a single-walled carbon nanotube (CNT) was examined via diffraction and spectroscopic techniques using both X-rays and electron beams. This new type of molybdenum oxide crystal has a chemical bonding state of MoO3, as confirmed by X-ray absorption spectroscopy, and the MoO3 units exhibit axial symmetry, as clarified by electron diffraction from bundled and individual CNTs encapsulating the crystal. To obtain three-dimensional information on the structure, a cross-sectional sample was prepared using a conventional dimple and ion-mill method. High-resolution transmission electron microscopy images exhibit ring-like shapes that originated from the arrangement of the MoO3 units inside the CNTs, as observed along the tube axis. The units are spaced 0.36 nm from each other in a ring arrangement and the distance between each ring is 0.391 nm.