Inference in artificial intelligence with deep optics and photonics.

Artificial intelligence tasks across numerous applications require accelerators for fast and low-power execution. Optical computing systems may be able to meet these domain-specific needs but, despite half a century of research, general-purpose optical computing systems have yet to mature into a practical technology. Artificial intelligence inference, however, especially for visual computing applications, may offer opportunities for inference based on optical and photonic systems. In this Perspective, we review recent work on optical computing for artificial intelligence applications and discuss its promise and challenges.

Volume holograms with linear diffraction efficiency relation by (3 + 1)D printing.

We demonstrate the fabrication of volume holograms using two-photon polymerization with dynamic control of light exposure. We refer to our method as (3 + 1)D printing. Volume holograms that are recorded by interfering reference and signal beams have a diffraction efficiency relation that is inversely proportional to the square of the number of superimposed holograms. By using (3 + 1)D printing for fabrication, the refractive index of each voxel is created independently and thus, by digitally filtering the undesired interference terms, the diffraction efficiency is now inversely proportional to the number of multiplexed gratings. We experimentally demonstrated this linear dependence by recording M = 50 volume gratings. To the best of our knowledge, this is the first experimental demonstration of distributed volume holograms that overcome the 1/M2 limit.

Forward-forward training of an optical neural network.

Neural networks (NNs) have demonstrated remarkable capabilities in various tasks, but their computation-intensive nature demands faster and more energy-efficient hardware implementations. Optics-based platforms, using technologies such as silicon photonics and spatial light modulators, offer promising avenues for achieving this goal. However, training multiple programmable layers together with these physical systems poses challenges, as they are difficult to fully characterize and describe with differentiable functions, hindering the use of error backpropagation algorithm. The recently introduced forward-forward algorithm (FFA) eliminates the need for perfect characterization of the physical learning system and shows promise for efficient training with large numbers of programmable parameters. The FFA does not require backpropagating an error signal to update the weights, rather the weights are updated by only sending information in one direction. The local loss function for each set of trainable weights enables low-power analog hardware implementations without resorting to metaheuristic algorithms or reinforcement learning. In this paper, we present an experiment utilizing multimode nonlinear wave propagation in an optical fiber demonstrating the feasibility of the FFA approach using an optical system. The results show that incorporating optical transforms in multilayer NN architectures trained with the FFA can lead to performance improvements, even with a relatively small number of trainable weights. The proposed method offers a new path to the challenge of training optical NNs and provides insights into leveraging physical transformations for enhancing the NN performance.

Photonic waveguide bundles using 3D laser writing and deep neural network image reconstruction.

In recent years, three-dimensional (3D) printing with multi-photon laser writing has become an essential tool for the manufacturing of three-dimensional optical elements. Single-mode optical waveguides are one of the fundamental photonic components, and are the building block for compact multicore fiber bundles, where thousands of single-mode elements are closely packed, acting as individual pixels and delivering the local information to a sensor. In this work, we present the fabrication of polymer rectangular step-index (STIN) optical waveguide bundles in the IP-Dip photoresist, using a commercial 3D printer. Moreover, we reduce the core-to-core spacing of the imaging bundles by means of a deep neural network (DNN) which has been trained with a large synthetic dataset, demonstrating that the scrambling of information due to diffraction and cross-talk between fiber cores can be undone. The DNN-based approach can be adopted in applications such as on-chip platforms and microfluidic systems where accurate imaging from in-situ printed fiber bundles suffer cross-talk. In this respect, we provide a design and fabrication guideline for such scenarios by employing the DNN not only as a post-processing technique but also as a design optimization tool.

3D reconstruction of weakly scattering objects from 2D intensity-only measurements using the Wolf transform.

A new approach to optical diffraction tomography (ODT) based on intensity measurements is presented. By applying the Wolf transform directly to intensity measurements, we observed unexpected behavior in the 3D reconstruction of the sample. Such a reconstruction does not explicitly represent a quantitative measure of the refractive index of the sample; however, it contains interesting qualitative information. This 3D reconstruction exhibits edge enhancement and contrast enhancement for nanostructures compared with the conventional 3D refractive index reconstruction and thus could be used to localize nanoparticles such as lipids inside a biological sample.

Fluorescence-Based and Fluorescent Label-Free Characterization of Polymer Nanoparticle Decorated T Cells.

Cells are attractive carriers for the transport and delivery of nanoparticulate cargo. The use of cell-based carriers allows one to enhance control over the biodistribution of drug-loaded polymers and polymer nanoparticles. One key element in the development of cell-based delivery systems is the loading of the cell-based carrier with the nanoparticle cargo, which can be achieved either by internalization of the payload or by immobilization on the cell surface. The surface modification of cells with nanoparticles or the internalization of nanoparticles by cells is usually monitored with fluorescence-based techniques, such as flow cytometry and confocal microscopy. In spite of the widespread use of these techniques, the use of fluorescent labels also poses some risks and has several drawbacks. Fluorescent dyes may bleach, or leach from, the nanoparticles or alter the physicochemical properties of nanoparticles and their interactions with and uptake by cells. Using poly(d,l-lactic acid) nanoparticles that are loaded with Coumarin 6, BODIPY 493/503, or DiO dyes as a model system, this paper demonstrates that the use of physically entrapped fluorescent labels can lead to false negative or erroneous results. The use of nanoparticles that contain covalently tethered fluorescent dyes instead was found to provide a robust approach to monitor cell surface conjugation reactions and to quantitatively analyze nanoparticle-decorated cells. Finally, it is shown that optical diffraction tomography is an attractive, alternative technique for the characterization of nanoparticle-decorated cells, which obviates the need for fluorescent labels.

All-fiber spatiotemporally mode-locked laser with multimode fiber-based filtering.

We demonstrate the first all-fiber multimode spatiotemporally mode-locked laser. The oscillator generates dissipative soliton pulses at 1036 nm with 12 mW average power, 6.24 ps duration, and 24.3 MHz repetition rate. The reported pulse energy (0.5 nJ) represents ∼4 times improvement over the previously reported single-mode all-normal dispersion mode-locked lasers with multimode interference-based filtering. Numerical simulations are performed to investigate the cavity and spatiotemporal mode-locking dynamics. The all-fiber oscillator we present shows promise for practical use since it can be fabricated simply.

Raman imaging through multimode sapphire fiber.

We report on a sapphire fiber Raman imaging probe's use for challenging applications where access is severely restricted. Small-dimension Raman probes have been developed previously for various clinical applications because they show great capability for diagnosing disease states in bodily fluids, cells, and tissues. However, applications of these sub-millimeter diameter Raman probes were constrained by two factors: first, it is difficult to incorporate filters and focusing optics at such small scale; second, the weak Raman signal is often obscured by strong background noise from the fiber probe material, especially the most commonly used silica, which has a strong broad background noise in low wavenumbers (<500-1700 cm-1). Here, we demonstrate the thinnest-known imaging Raman probe with a 60 µm diameter Sapphire multimode fiber in which both excitation and signal collection pass through. This probe takes advantage of the low fluorescence and narrow Raman peaks of Sapphire, its inherent high temperature and corrosion resistance, and large numerical aperture (NA). Raman images of Polystyrene beads, carbon nanotubes, and CaSO4 agglomerations are obtained with a spatial resolution of 1 µm and a field of view of 30 µm. Our imaging results show that single polystyrene bead (~15 µm diameter) can be differentiated from a mixture with CaSO4 agglomerations, which has a close Raman shift.

Single-photon three-dimensional microfabrication through a multimode optical fiber.

Two-photon polymerization (TPP) processes have enabled the fabrication of advanced and functional microstructures. However, most TPP platforms are bulky and require the use of expensive femtosecond lasers. Here, we propose an inexpensive and compact alternative to TPP by adapting an endoscopic imaging system for single-photon three-dimensional microfabrication. The wavefront of a visible continuous-wave laser beam is shaped so that it focuses into a photoresist through a 5 cm long ultra-thin multimode optical fiber (∅70 µm, NA 0.64). Using this device, we show that single-photon polymerization can be confined to the phase-controlled focal spot thanks to the non-linearity of the photoresist, likely due to oxygen radical scavenging. Thus, by exploiting this non-linearity with a specific overcuring method we demonstrate single-photon three-dimensional fabrication of solid and hollow microstructures through a multimode fiber with a 1.0-µm lateral and 21.5-µm axial printing resolution. This opens up new possibilities for advanced and functional microfabrication through endoscopic probes with inexpensive laser sources.

Versatile reconstruction framework for diffraction tomography with intensity measurements and multiple scattering.

Taking benefit from recent advances in both phase retrieval and estimation of refractive indices from holographic measurements, we propose a unified framework to reconstruct them from intensity-only measurements. Our method relies on a generic and versatile formulation of the inverse problem and includes sparsity constraints. Its modularity enables the use of a variety of forward models, from simple linear ones to more sophisticated nonlinear ones, as well as various regularizers. We present reconstructions that deploy either the beam-propagation method or the iterative Lippmann-Schwinger model, combined with total-variation regularization. They suggest that our proposed (intensity-only) method can reach the same performance as reconstructions from holographic (complex) data. This is of particular interest from a practical point of view because it allows one to simplify the acquisition setup.

Bend translation in multimode fiber imaging.

Light propagation in multimode fibers is typically assumed to be extremely sensitive to changes in geometry. We study here a particular configuration where an S-shaped bend is translated between two sections of fiber. In this sliding bend configuration, we show that nearly constant propagation characteristics can be obtained in certain fibers. Several fibers were tested using a bend with a peak radius of curvature of 25 mm. We found large differences in bending behavior between fibers of varying core diameters and numerical apertures. Fibers with a large numerical aperture are found to be more stable. In several fibers, the bend can be translated over a distance of 25 mm with a limited impact on imaging performance. The experimental results are confirmed using simulations. Our findings shed a new light on bending sensitivity in multimode fibers, and open up more possibilities for their use as imaging devices.

High power, ultrashort pulse control through a multi-core fiber for ablation.

Ultrashort pulse ablation has become a useful tool for micromachining and biomedical surgical applications. Implementation of ultrashort pulse ablation in confined spaces has been limited by endoscopic delivery and focusing of a high peak power pulse. Here we demonstrate ultrashort pulse ablation through a thin multi-core fiber (MCF) using wavefront shaping, which allows for focusing and scanning the pulse without requiring distal end optics and enables a smaller ablation tool. The intensity necessary for ablation is significantly higher than for multiphoton imaging. We show that the ultimate limitations of the MCF based ablation are the nonlinear effects induced by the pulse in the MCFs cores. We characterize and compare the performance of two devices utilizing a different number of cores and demonstrate ultrashort pulse ablation on a thin film of gold.

Three-dimensional microfabrication through a multimode optical fiber.

3D printing based on additive manufacturing is an advanced manufacturing technique that allows the fabrication of arbitrary macroscopic and microscopic objects. Many 3D printing systems require large optical elements or nozzles in proximity to the built structure. This prevents their use in applications in which there is no direct access to the area where the objects have to be printed. Here, we demonstrate three-dimensional microfabrication based on two-photon polymerization (TPP) through an ultra-thin printing nozzle of 560 µm in diameter. Using wavefront shaping, femtosecond infrared pulses are focused and scanned through a multimode optical fiber (MMF) inside a photoresist that polymerizes via two-photon absorption. We show the construction of arbitrary 3D structures built with voxels of diameters down to 400 nm on the other side of the fiber. To our knowledge, this is the first demonstration of microfabrication through a multimode optical fiber. The proposed printing nozzle can reach and manufacture micro-structures in otherwise inaccessible areas through small apertures. Our work represents a new area which we refer to as endofabrication.

Hollow Mesoporous Plasmonic Nanoshells for Enhanced Solar Vapor Generation.

In the past decade, nanomaterials have made their way into a variety of technologies in solar energy, enhancing the performance by taking advantage of the phenomena inherent to the nanoscale. Recent examples exploit plasmonic core/shell nanoparticles to achieve efficient direct steam generation, showing great promise of such nanoparticles as a useful material for solar applications. In this paper, we demonstrate a novel technique for fabricating bimetallic hollow mesoporous plasmonic nanoshells that yield a higher solar vapor generation rate compared with their solid-core counterparts. On the basis of a combination of nanomasking and incomplete galvanic replacement, the hollow plasmonic nanoshells can be fabricated with tunable absorption and minimized scattering. When exposed to sun light, each hollow nanoshell generates vapor bubbles simultaneously from the interior and exterior. The vapor nucleating from the interior expands and diffuses through the pores and combines with the bubbles formed on the outer wall. The lack of a solid core significantly accelerates the initial vapor nucleation and the overall steam generation dynamics. More importantly, because the density of the hollow porous nanoshells is essentially equal to the surrounding host medium these particles are much less prone to sedimentation, a problem that greatly limits the performance and implementation of standard nanoparticle dispersions.

Calibration-free imaging through a multicore fiber using speckle scanning microscopy.

The images produced by multicore endoscopes are pixelated, and their resolution is limited by the core-to-core spacing. Lenses can be used to improve the resolution, but this reduces the field of view proportionally. Lensless endoscopy through multicore fibers can be achieved by using wavefront shaping techniques. This requires a calibration step, and the conformation of the fiber must remain constant over time. Here we demonstrate that, without a calibration step and in the presence of core-to-core coupling, we can obtain fluorescence images with a resolution better than the core-to-core spacing. This is accomplished by taking advantage of the memory effect present in these kinds of fibers.

Light control in a multicore fiber using the memory effect.

Endoscopic imaging through a multicore fiber (MCF) is widely used but is affected by pixelated images, which limits its resolution to a few micrometers. This is due to the spacing between the cores in the MCF, which is large enough to avoid core-to-core light coupling. Wavefront shaping techniques have been shown to focus light to a resolution finer than the inter core spacing, however a long calibration procedure is needed. Moreover the calibration depends on the optical fiber conformation. Here, we show a calibration method using only one digital hologram. The method is based on digital phase conjugation and the memory effect of the MCF to focus and scan a spot. In addition, we show how simple patterns can be projected using the same multicore fiber.

Two-photon imaging through a multimode fiber.

In this work we demonstrate 3D imaging using two-photon excitation through a 20 cm long multimode optical fiber (MMF) of 350 µm diameter. The imaging principle is similar to single photon fluorescence through a MMF, except that a focused femtosecond pulse is delivered and scanned over the sample. In our approach, focusing and scanning through the fiber is accomplished by digital phase conjugation using mode selection by time gating with an ultra-fast reference pulse. The excited two-photon emission is collected through the same fiber. We demonstrate depth sectioning by scanning the focused pulse in a 3D volume over a sample consisting of fluorescent beads suspended in a polymer. The achieved resolution is 1 µm laterally and 15 µm axially. Scanning is performed over an 80x80 µm field of view. To our knowledge, this is the first demonstration of high-resolution three-dimensional imaging using two-photon fluorescence through a multimode fiber.

Enhanced resolution in a multimode fiber imaging system.

Multimode fibers have recently been demonstrated to be a promising candidate for ultrathin and high resolution endoscopy. However, this method does not offer depth discrimination for fluorescence imaging and the numerical aperture of the fiber limits its resolution. In this paper we demonstrate optical sectioning and enhanced resolution using saturated excitation and temporal modulation. Using a continuous wave laser excitation, we demonstrate improved resolution in all three dimensions and increased image contrast by rejecting out of focus light.

Improving the quality of filament-impaired images in Kerr media by statistical averaging.

In focusing Kerr media, small-scale filamentation is the major obstacle to imaging at high light intensities. In this article, we experimentally and numerically demonstrate a method based on statistical averaging to reduce the detrimental effects of filamentation on the reconstructed images. The experiments are performed with femtosecond optical pulses propagating through a nonlinear liquid (toluene). We use digital holography to capture the transmitted optical image. The reverse propagation of the captured field is numerically performed using a numerical solution of the nonlinear Schrödinger equation. Because of their intrinsic sensitivity to measurement noise, filaments fail to propagate back on their initial trajectories and parasitic filaments form. The principle of the method is the introduction of artificial perturbations on the measurement, which spatially displace the parasitic filaments. By averaging the reconstruction over many realizations of the artificial perturbation, we show that the reconstruction improves the quality of the images. Finally, in order to identify the different regimes of optical power for which the filaments are time reversible, we also derive an analytical estimate for the condition number of the nonlinear propagator.

Digital confocal microscopy through a multimode fiber.

Acquiring high-contrast optical images deep inside biological tissues is still a challenging problem. Confocal microscopy is an important tool for biomedical imaging since it improves image quality by rejecting background signals. However, it suffers from low sensitivity in deep tissues due to light scattering. Recently, multimode fibers have provided a new paradigm for minimally invasive endoscopic imaging by controlling light propagation through them. Here we introduce a combined imaging technique where confocal images are acquired through a multimode fiber. We achieve this by digitally engineering the excitation wavefront and then applying a virtual digital pinhole on the collected signal. In this way, we are able to acquire images through the fiber with significantly increased contrast. With a fiber of numerical aperture 0.22, we achieve a lateral resolution of 1.5µm, and an axial resolution of 12.7µm. The point-scanning rate is currently limited by our spatial light modulator (20Hz).