Towards real-time photorealistic 3D holography with deep neural networks.

The ability to present three-dimensional (3D) scenes with continuous depth sensation has a profound impact on virtual and augmented reality, human-computer interaction, education and training. Computer-generated holography (CGH) enables high-spatio-angular-resolution 3D projection via numerical simulation of diffraction and interference1. Yet, existing physically based methods fail to produce holograms with both per-pixel focal control and accurate occlusion2,3. The computationally taxing Fresnel diffraction simulation further places an explicit trade-off between image quality and runtime, making dynamic holography impractical4. Here we demonstrate a deep-learning-based CGH pipeline capable of synthesizing a photorealistic colour 3D hologram from a single RGB-depth image in real time. Our convolutional neural network (CNN) is extremely memory efficient (below 620 kilobytes) and runs at 60 hertz for a resolution of 1,920 × 1,080 pixels on a single consumer-grade graphics processing unit. Leveraging low-power on-device artificial intelligence acceleration chips, our CNN also runs interactively on mobile (iPhone 11 Pro at 1.1 hertz) and edge (Google Edge TPU at 2.0 hertz) devices, promising real-time performance in future-generation virtual and augmented-reality mobile headsets. We enable this pipeline by introducing a large-scale CGH dataset (MIT-CGH-4K) with 4,000 pairs of RGB-depth images and corresponding 3D holograms. Our CNN is trained with differentiable wave-based loss functions5 and physically approximates Fresnel diffraction. With an anti-aliasing phase-only encoding method, we experimentally demonstrate speckle-free, natural-looking, high-resolution 3D holograms. Our learning-based approach and the Fresnel hologram dataset will help to unlock the full potential of holography and enable applications in metasurface design6,7, optical and acoustic tweezer-based microscopic manipulation8-10, holographic microscopy11 and single-exposure volumetric 3D printing12,13.

Airy-beam holographic sonogenetics for advancing neuromodulation precision and flexibility.

Advancing our understanding of brain function and developing treatments for neurological diseases hinge on the ability to modulate neuronal groups in specific brain areas without invasive techniques. Here, we introduce Airy-beam holographic sonogenetics (AhSonogenetics) as an implant-free, cell type-specific, spatially precise, and flexible neuromodulation approach in freely moving mice. AhSonogenetics utilizes wearable ultrasound devices manufactured using 3D-printed Airy-beam holographic metasurfaces. These devices are designed to manipulate neurons genetically engineered to express ultrasound-sensitive ion channels, enabling precise modulation of specific neuronal populations. By dynamically steering the focus of Airy beams through ultrasound frequency tuning, AhSonogenetics is capable of modulating neuronal populations within specific subregions of the striatum. One notable feature of AhSonogenetics is its ability to flexibly stimulate either the left or right striatum in a single mouse. This flexibility is achieved by simply switching the acoustic metasurface in the wearable ultrasound device, eliminating the need for multiple implants or interventions. AhSonogentocs also integrates seamlessly with in vivo calcium recording via fiber photometry, showcasing its compatibility with optical modalities without cross talk. Moreover, AhSonogenetics can generate double foci for bilateral stimulation and alleviate motor deficits in Parkinson's disease mice. This advancement is significant since many neurological disorders, including Parkinson's disease, involve dysfunction in multiple brain regions. By enabling precise and flexible cell type-specific neuromodulation without invasive procedures, AhSonogenetics provides a powerful tool for investigating intact neural circuits and offers promising interventions for neurological disorders.

Detecting material state changes in the nucleolus by label-free digital holographic microscopy.

Ribosome biogenesis is initiated in the nucleolus, a multiphase biomolecular condensate formed by liquid-liquid phase separation. The nucleolus is a powerful disease biomarker and stress biosensor whose morphology reflects function. Here we have used digital holographic microscopy (DHM), a label-free quantitative phase contrast microscopy technique, to detect nucleoli in adherent and suspension human cells. We trained convolutional neural networks to detect and quantify nucleoli automatically on DHM images. Holograms containing cell optical thickness information allowed us to define a novel index which we used to distinguish nucleoli whose material state had been modulated optogenetically by blue-light-induced protein aggregation. Nucleoli whose function had been impacted by drug treatment or depletion of ribosomal proteins could also be distinguished. We explored the potential of the technology to detect other natural and pathological condensates, such as those formed upon overexpression of a mutant form of huntingtin, ataxin-3, or TDP-43, and also other cell assemblies (lipid droplets). We conclude that DHM is a powerful tool for quantitatively characterizing nucleoli and other cell assemblies, including their material state, without any staining.

Fast optical recording of neuronal activity by three-dimensional custom-access serial holography.

Optical recording of neuronal activity in three-dimensional (3D) brain circuits at cellular and millisecond resolution in vivo is essential for probing information flow in the brain. While random-access multiphoton microscopy permits fast optical access to neuronal targets in three dimensions, the method is challenged by motion artifacts when recording from behaving animals. Therefore, we developed three-dimensional custom-access serial holography (3D-CASH). Built on a fast acousto-optic light modulator, 3D-CASH performs serial sampling at 40 kHz from neurons at freely selectable 3D locations. Motion artifacts are eliminated by targeting each neuron with a size-optimized pattern of excitation light covering the cell body and its anticipated displacement field. Spike rates inferred from GCaMP6f recordings in visual cortex of awake mice tracked the phase of a moving bar stimulus with higher spike correlation between intra compared to interlaminar neuron pairs. 3D-CASH offers access to the millisecond correlation structure of in vivo neuronal activity in 3D microcircuits.

Quantitative label-free digital holographic imaging of cardiomyocyte optical volume, nucleation, and cell division.

Cardiac regeneration in newborn rodents depends on the ability of pre-existing cardiomyocytes to proliferate and divide. This capacity is lost within the first week of postnatal development when these cells rapidly switch from hyperplasia to hypertrophy, withdraw from the cell cycle, become binucleated, and increase in size. How these dynamic changes in cell size and nucleation impact cardiomyocyte proliferative potential is not well understood. In this study, we innovate the application of a commercially available digital holographic imaging microscope, the Holomonitor M4, to evaluate the proliferative responses of mononucleated and binucleated cardiomyocytes after CHIR99021 treatment, a model proliferative stimulus. This system enables long-term label-free quantitative tracking of primary cardiomyocyte dynamics in real-time with single-cell resolution. Our results confirm that chemical inhibition of glycogen synthase kinase 3 with CHIR99021 promotes complete cell division of both mononucleated and binucleated cardiomyocytes with high frequency. Quantitative tracking of cardiomyocyte volume dynamics during these proliferative events revealed that both mononucleated and binucleated cardiomyocytes reach a similar size-increase threshold prior to attempted cell division. Binucleated cardiomyocytes attempt to divide with lower frequency than mononucleated cardiomyocytes, which may be associated with inadequate increases in cell size. By defining the interrelationship between cardiomyocyte size, nucleation, and cell cycle control, we may better understand the cellular mechanisms that drive the loss of mammalian cardiac regenerative capacity after birth.

Label-free, high-throughput holographic imaging to evaluate mammalian gametes and embryos†.

Assisted reproduction is one of the significant tools to treat human infertility. Morphological assessment is the primary method to determine sperm and embryo viability during in vitro fertilization cycles. It has the advantage of being a quick, convenient, and inexpensive means of assessment. However, visual observation is of limited predictive value for early embryo morphology. It has led many to search for other imaging tools to assess the reproductive potential of a given embryo. The limitations of visual assessment apply to both humans and animals. One recent innovation in assisted reproduction technology imaging is interferometric phase microscopy, also known as holographic microscopy. Interferometric phase microscopy/quantitative phase imaging is the next likely progression of analytical microscopes for the assisted reproduction laboratory. The interferometric phase microscopy system analyzes waves produced by the light as it passes through the specimen observed. The microscope collects the light waves produced and uses the algorithm to create a hologram of the specimen. Recently, interferometric phase microscopy has been combined with quantitative phase imaging, which joins phase contrast microscopy with holographic microscopy. These microscopes collect light waves produced and use the algorithm to create a hologram of the specimen. Unlike other systems, interferometric phase microscopy can provide a quantitative digital image, and it can make 2D and 3D images of the samples. This review summarizes some newer and more promising quantitative phase imaging microscopy systems for evaluating gametes and embryos. Studies clearly show that quantitative phase imaging is superior to bright field microscopy-based evaluation methods when evaluating sperm and oocytes prior to IVF and embryos prior to transfer. However, further assessment of these systems for efficacy, reproducibility, cost-effectiveness, and embryo/gamete safety must take place before they are widely adopted.

Comparison of holotomographic microscopy and coherence-controlled holographic microscopy.

Quantitative phase imaging (QPI) is a powerful tool for label-free visualisation of living cells. Here, we compare two QPI microscopes - the Telight Q-Phase microscope and the Nanolive 3D Cell Explorer-fluo microscope. Both systems provide unbiased information about cell morphology, such as individual cell dry mass, perimeter and area. The Q-Phase microscope uses artefact-free, coherence-controlled holographic imaging technology to visualise cells in real time with minimal phototoxicity. The 3D Cell Explorer-fluo employs laser-based holotomography to reconstruct 3D images of living cells, visualising their internal structures and dynamics. Here, we analysed the strengths and limitations of both microscopes when examining two morphologically distinct cell lines - the cuboidal epithelial MDCK cells which form multicellular clusters and solitary growing Rat2 fibroblasts. We focus mainly on the ability of the devices to generate images suitable for single-cell segmentation by the built-in software, and we discuss the segmentation results and quantitative data generated from the segmented images. We show that both microscopes offer slightly different advantages, and the choice between them depends on the specific requirements and goals of the user.

Low-energy electron holography imaging of conformational variability of single-antibody molecules from electrospray ion beam deposition.

Imaging of proteins at the single-molecule level can reveal conformational variability, which is essential for the understanding of biomolecules. To this end, a biologically relevant state of the sample must be retained during both sample preparation and imaging. Native electrospray ionization (ESI) can transfer even the largest protein complexes into the gas phase while preserving their stoichiometry and overall shape. High-resolution imaging of protein structures following native ESI is thus of fundamental interest for establishing the relation between gas phase and solution structure. Taking advantage of low-energy electron holography's (LEEH) unique capability of imaging individual proteins with subnanometer resolution, we investigate the conformational flexibility of Herceptin, a monoclonal IgG antibody, deposited by native electrospray mass-selected ion beam deposition (ES-IBD) on graphene. Images reconstructed from holograms reveal a large variety of conformers. Some of these conformations can be mapped to the crystallographic structure of IgG, while others suggest that a compact, gas-phase-related conformation, adopted by the molecules during ES-IBD, is retained. We can steer the ratio of those two types of conformations by changing the landing energy of the protein on the single-layer graphene surface. Overall, we show that LEEH can elucidate the conformational heterogeneity of inherently flexible proteins, exemplified here by IgG antibodies, and thereby distinguish gas-phase collapse from rearrangement on surfaces.

Effect of sound-induced vibrations of the pinna on head-related transfer functions: Experimental and numerical investigations.

Numerical simulations of head-related transfer functions (HRTFs) conventionally assume a rigid boundary condition for the pinna. The human pinna, however, is an elastic deformable body that can vibrate due to incident acoustic waves. This work investigates how sound-induced vibrations of the pinna can affect simulated HRTF magnitudes. The work will motivate the research question by measuring the sound-induced vibrational patterns of an artificial pinna with a high-speed holographic interferometric system. Then, finite element simulations are used to determine HRTFs for a tabletop model of the B&K 5128 head and torso simulator for a number of directions. Two scenarios are explored: one where the pinna is modeled as perfectly rigid, and another where the pinna is modeled as linear elastic with material properties close to that of auricular cartilage. The findings suggest that pinna vibrations have negligible effects on HRTF magnitudes up to 5 kHz. The same conclusion, albeit with less certainty, is drawn for higher frequencies. Finally, the importance of the elastic domain's material properties is emphasized and possible implications for validation studies on dummy heads 1as well as the limitations of the present work are discussed in detail.

Live Cell Imaging by Single-Shot Common-Path Wide Field-of-View Reflective Digital Holographic Microscope.

Quantitative phase imaging by digital holographic microscopy (DHM) is a nondestructive and label-free technique that has been playing an indispensable role in the fields of science, technology, and biomedical imaging. The technique is competent in imaging and analyzing label-free living cells and investigating reflective surfaces. Herein, we introduce a new configuration of a wide field-of-view single-shot common-path off-axis reflective DHM for the quantitative phase imaging of biological cells that leverages several advantages, including being less-vibration sensitive to external perturbations due to its common-path configuration, also being compact in size, simple in optical design, highly stable, and cost-effective. A detailed description of the proposed DHM system, including its optical design, working principle, and capability for phase imaging, is presented. The applications of the proposed system are demonstrated through quantitative phase imaging results obtained from the reflective surface (USAF resolution test target) as well as transparent samples (living plant cells). The proposed system could find its applications in the investigation of several biological specimens and the optical metrology of micro-surfaces.

Label-Free Three-Dimensional Morphological Characterization of Cell Death Using Holographic Tomography.

This study presents a novel label-free approach for characterizing cell death states, eliminating the need for complex molecular labeling that may yield artificial or ambiguous results due to technical limitations in microscope resolution. The proposed holographic tomography technique offers a label-free avenue for capturing precise three-dimensional (3D) refractive index morphologies of cells and directly analyzing cellular parameters like area, height, volume, and nucleus/cytoplasm ratio within the 3D cellular model. We showcase holographic tomography results illustrating various cell death types and elucidate distinctive refractive index correlations with specific cell morphologies complemented by biochemical assays to verify cell death states. These findings hold promise for advancing in situ single cell state identification and diagnosis applications.

Holo-Stroke: Assessing for Immersive Stroke Care Through Stroke Hologram Teleportation.

Background: Augmented reality enables the wearer to see both their physical environment and virtual objects. Holograms could allow 3D video of providers to be transmitted to distant sites, allowing patients to interact with virtual providers as if they are in the same physical space. Our aim was to determine if Tele-Stroke augmented with Holo-Stroke, compared with Tele-Stroke alone, could improve satisfaction and perception of immersion for the patient. Methods: Kinect cameras positioned at 90-degree intervals around the hub practitioner were used. Cameras streamed real-time optical video to a unity point-cloud program where the data were stitched together in a 360-degree view. The resultant hologram was positioned in 3D space and was visible through the head-mounted display by the patient. Radiology images were shared in Tele-Stroke and via hologram. Likert satisfaction questions were administered. Wilcoxon signed-rank testing was used. Results: Each of the 30 neurology clinic participants scored both Tele-Stroke and Holo-Stroke. Out of these, 29 patients completed the assessments (1 failure owing to computer reboot). Average age was 52 years, with 53.3% of the patients being female, 70.0% being White, and 13.3% being Hispanic. Likert scale score median "Overall" was 32 Tele-Stroke versus 48 Holo-Stroke (p < 0.00001), "Immersion" was 5 versus 10 (p < 0.00001), "Beneficial Technique" was 6 versus 10 (p < 0.00001), and "Ability to See Images" was 5 versus 10 (p < 0.00001). Discussion: Holo-Stroke 3D holographic Tele-Stroke exams resulted in feasibility, satisfaction, and high perception of immersion for the patient. Patients were enthusiastic for the more immersive, personal discussion with their provider and a robust way to experience radiology images. Though further assessments are needed, Holo-Stroke can help the provider "be there, not just see there!"

Three-Dimensional Holographic-Guided Robotic Lung Segmentectomy for Deep Pulmonary Nodules: Technique and Initial Results.

Background: Diagnosis and treatment of small and isolated lung nodules remain challenging issues. Purpose: The aim of this article is to report the technique of real-time navigation using holographic reconstruction technology combined with a robot assisted thoracic surgery (RATS) platform for lung resection in patients with small deep nodules.Research Design: The pre-surgery 3D planning was based on the chest CT scan. The reconstruction was uploaded to a head-mounted display for real-time navigation during mini invasive robot assisted surgery performed with an open console platform. We evaluated this technique with the success rate of diagnosis, the operative time and the post-operative course.Study Sample: This technique was performed in 6 patients (4 female, mean age 65 years) to date.Results: The precision of the head-mounted display based localization system was effective in all cases without the need of open conversion. The mean diameter of the nodules was 8 mm (6-9). The diagnosis was a lung cancer (n = 5) and tuberculoma (n = 1). The mean operative time was 125 min (100-145). The mean hospital stay was 2.5 days (1-3).Conclusions: In conclusion, the intraoperative navigation using the 3D holographic assistance was an helpful tool for mini invasive RATS lung segmentectomy without the need of preoperative localization.

[Simulation and Navigation in the Neurosurgical Field Using Three-Dimensional Hholograms by Mixed Reality Devices].

Recently, three-dimensional(3D)holograms from mixed-reality(MR)devices have become available in the medical field. 3D holographic images can provide immersive and intuitive information that has been reported to be very useful for preoperative simulations. Compared with conventional 3D images on a two-dimensional(2D)monitor, 3D holograms offer a higher level of realism, allowing observation of the images anytime and anywhere if the MR device is operational. Even during surgery, surgeons can check realistic 3D holograms in front of them, above the surgical field, without having to turn their heads toward a 2D monitor on the wall. 3D holograms can also be used for neuronavigation if the hologram is tracked to the patient's real head. This method can be defined as 3D augmented reality(AR)navigation, which shows a hologram of a target, such as a tumor or aneurysm, inside the head and brain. In the future, interventions using these techniques with 3D holograms from MR devices are expected to evolve and develop new types of treatments for endoscopic surgery or fluoroscopy-guided endovascular surgery.

Development of serial X-ray fluorescence holography for radiation-sensitive protein crystals.

X-ray fluorescence holography (XFH) is a powerful atomic resolution technique capable of directly imaging the local atomic structure around atoms of a target element within a material. Although it is theoretically possible to use XFH to study the local structures of metal clusters in large protein crystals, the experiment has proven difficult to perform, especially on radiation-sensitive proteins. Here, the development of serial X-ray fluorescence holography to allow the direct recording of hologram patterns before the onset of radiation damage is reported. By combining a 2D hybrid detector and the serial data collection used in serial protein crystallography, the X-ray fluorescence hologram can be directly recorded in a fraction of the measurement time needed for conventional XFH measurements. This approach was demonstrated by obtaining the Mn Kα hologram pattern from the protein crystal Photosystem II without any X-ray-induced reduction of the Mn clusters. Furthermore, a method to interpret the fluorescence patterns as real-space projections of the atoms surrounding the Mn emitters has been developed, where the surrounding atoms produce large dark dips along the emitter-scatterer bond directions. This new technique paves the way for future experiments on protein crystals that aim to clarify the local atomic structures of their functional metal clusters, and for other related XFH experiments such as valence-selective XFH or time-resolved XFH.

On monocytes and lymphocytes biolens clustering by in flow holographic microscopy.

Live cells act as biological lenses and can be employed as real-world optical components in bio-hybrid systems. Imaging at nanoscale, optical tweezers, lithography and also photonic waveguiding are some of the already proven functionalities, boosted by the advantage that cells are fully biocompatible for intra-body applications. So far, various cell types have been studied for this purpose, such as red blood cells, bacterial cells, stem cells and yeast cells. White Blood Cells (WBCs) play a very important role in the regulation of the human body activities and are usually monitored for assessing its health. WBCs can be considered bio-lenses but, to the best of our knowledge, characterization of their optical properties have not been investigated yet. Here, we report for the first time an accurate study of two model classes of WBCs (i.e., monocytes and lymphocytes) by means of a digital holographic microscope coupled with a microfluidic system, assuming WBCs bio-lens characteristics. Thus, quantitative phase maps for many WBCs have been retrieved in flow-cytometry (FC) by achieving a significant statistical analysis to prove the enhancement in differentiation among sphere-like bio-lenses according to their sizes (i.e., diameter d) exploiting intensity parameters of the modulated light in proximity of the cell optical axis. We show that the measure of the low intensity area (S: I z < I th z ) in a fixed plane, is a feasible parameter for cell clustering, while achieving robustness against experimental misalignments and allowing to adjust the measurement sensitivity in post-processing. 2D scatterplots of the identified parameters (d-S) show better differentiation respect to the 1D case. The results show that the optical focusing properties of WBCs allow the clustering of the two populations by means of a mere morphological analysis, thus leading to the new concept of cell-optical-fingerprint avoiding fluorescent dyes. This perspective can open new routes in biomedical sciences, such as the chance to find optical-biomarkers at single cell level for label-free diagnosis.

Fast 2-photon stimulation using holographic patterns.

Two decades after its introduction, optogenetics - a biological technique to control the activity of neurons or other cell types with light - remains a cutting edge and promising tool to study biological processes. Its increasing usage in research varies widely from causally exploring biological mechanisms and neural computations, to neurostimulation and sensory restauration. To stimulate neurons in the brain, a variety of approaches have been developed to generate precise spatiotemporal light patterns. Yet certain constrains still exists in the current optical techniques to activate a neuronal population with both cellular resolution and millisecond precision. Here, we describe an experimental setup allowing to stimulate a few tens of neurons in a plane at sub-millisecond rates using 2-photon activation. A liquid crystal on silicon spatial light modulator (LCoS-SLM) was used to generate spatial patterns in 2 dimensions. The image of the patterns was formed on the plane of a digital micromirror device (DMD) that was used as a fast temporal modulator of each region of interest. Using fluorescent microscopy and patch-clamp recording of neurons in culture expressing the light-gated ion channels, we characterized the temporal and spatial resolution of the microscope. We described the advantages of combining the LCoS-SLM with the DMD to maximize the temporal precision, modulate the illumination amplitude, and reduce background activation. Finally, we showed that this approach can be extended to patterns in 3 dimensions. We concluded that the methodology is well suited to address important questions about the role of temporal information in neuronal coding.

Aberration-free holographic microscope for simultaneous imaging and stimulation of neuronal populations.

A technical challenge in neuroscience is to record and specifically manipulate the activity of neurons in living animals. This can be achieved in some preparations with two-photon calcium imaging and photostimulation. These methods can be extended to three dimensions by holographic light sculpting with spatial light modulators (SLMs). At the same time, performing simultaneous holographic imaging and photostimulation is still cumbersome, requiring two light paths with separate SLMs. Here we present an integrated optical design using a single SLM for simultaneous imaging and photostimulation. Furthermore, we applied axially dependent adaptive optics to make the system aberration-free, and developed software for calibrations and closed-loop neuroscience experiments. Finally, we demonstrate the performance of the system with simultaneous calcium imaging and optogenetics in mouse primary auditory cortex in vivo. Our integrated holographic system could facilitate the systematic investigation of neural circuit function in awake behaving animals.

Functionally integrated waveguide illuminator for common-path digital holographic microscopy through random media.

We designed and fabricated a functionally integrated optical waveguide illuminator specially for common-path digital holographic microscopy through random media. The waveguide illuminator creates two point sources with desired phase shifts, which are located close to one another so that the common-path condition of the object and reference illumination is satisfied. Thereby, the proposed device permits phase-shift digital holographic microscopy free from bulky optical elements such as a beam splitter, an objective lens, and a piezoelectric transducer for phase shifting. Using the proposed device, microscopic 3D imaging through a highly heterogeneous double-composite random medium was experimentally demonstrated by means of common-path phase-shift digital holography.

Polarimetric calibrated robust dual-SLM complex-amplitude computer-generated holography.

Liquid crystal on silicon (LCoS) is a widely used spatial light modulator (SLM) in computer-generated holography (CGH). However, the phase-modulating profile of LCoS is often not ideally uniform in application, bringing about undesired intensity fringes. In this study, we overcome this problem by proposing a highly robust dual-SLM complex-amplitude CGH technique, which incorporates a polarimetric mode and a diffractive mode. The polarimetric mode linearizes the general phase modulations of the two SLMs separately, while the diffractive mode uses camera-in-the-loop optimization to achieve improved holographic display. Experimental results show the effectiveness of our proposal in improving reconstructing accuracy by 21.12% in peak signal-to-noise ratio (PSNR) and 50.74% in structure similarity index measure (SSIM), using LCoS SLMs with originally non-uniform phase-modulating profiles.