An actor-model framework for visual sensory encoding.

A fundamental challenge in neuroengineering is determining a proper artificial input to a sensory system that yields the desired perception. In neuroprosthetics, this process is known as artificial sensory encoding, and it holds a crucial role in prosthetic devices restoring sensory perception in individuals with disabilities. For example, in visual prostheses, one key aspect of artificial image encoding is to downsample images captured by a camera to a size matching the number of inputs and resolution of the prosthesis. Here, we show that downsampling an image using the inherent computation of the retinal network yields better performance compared to learning-free downsampling methods. We have validated a learning-based approach (actor-model framework) that exploits the signal transformation from photoreceptors to retinal ganglion cells measured in explanted mouse retinas. The actor-model framework generates downsampled images eliciting a neuronal response in-silico and ex-vivo with higher neuronal reliability than the one produced by a learning-free approach. During the learning process, the actor network learns to optimize contrast and the kernel's weights. This methodological approach might guide future artificial image encoding strategies for visual prostheses. Ultimately, this framework could be applicable for encoding strategies in other sensory prostheses such as cochlear or limb.

Volume holograms with linear diffraction efficiency relation by (3 + 1)D printing.

We demonstrate the fabrication of volume holograms using two-photon polymerization with dynamic control of light exposure. We refer to our method as (3 + 1)D printing. Volume holograms that are recorded by interfering reference and signal beams have a diffraction efficiency relation that is inversely proportional to the square of the number of superimposed holograms. By using (3 + 1)D printing for fabrication, the refractive index of each voxel is created independently and thus, by digitally filtering the undesired interference terms, the diffraction efficiency is now inversely proportional to the number of multiplexed gratings. We experimentally demonstrated this linear dependence by recording M = 50 volume gratings. To the best of our knowledge, this is the first experimental demonstration of distributed volume holograms that overcome the 1/M2 limit.

Beyond fluorescence: advances in computational label-free full specificity in 3D quantitative phase microscopy.

Despite remarkable progresses in quantitative phase imaging (QPI) microscopes, their wide acceptance is limited due to the lack of specificity compared with the well-established fluorescence microscopy. In fact, the absence of fluorescent tag prevents to identify subcellular structures in single cells, making challenging the interpretation of label-free 2D and 3D phase-contrast data. Great effort has been made by many groups worldwide to address and overcome such limitation. Different computational methods have been proposed and many more are currently under investigation to achieve label-free microscopic imaging at single-cell level to recognize and quantify different subcellular compartments. This route promises to bridge the gap between QPI and FM for real-world applications.

Forward-forward training of an optical neural network.

Neural networks (NNs) have demonstrated remarkable capabilities in various tasks, but their computation-intensive nature demands faster and more energy-efficient hardware implementations. Optics-based platforms, using technologies such as silicon photonics and spatial light modulators, offer promising avenues for achieving this goal. However, training multiple programmable layers together with these physical systems poses challenges, as they are difficult to fully characterize and describe with differentiable functions, hindering the use of error backpropagation algorithm. The recently introduced forward-forward algorithm (FFA) eliminates the need for perfect characterization of the physical learning system and shows promise for efficient training with large numbers of programmable parameters. The FFA does not require backpropagating an error signal to update the weights, rather the weights are updated by only sending information in one direction. The local loss function for each set of trainable weights enables low-power analog hardware implementations without resorting to metaheuristic algorithms or reinforcement learning. In this paper, we present an experiment utilizing multimode nonlinear wave propagation in an optical fiber demonstrating the feasibility of the FFA approach using an optical system. The results show that incorporating optical transforms in multilayer NN architectures trained with the FFA can lead to performance improvements, even with a relatively small number of trainable weights. The proposed method offers a new path to the challenge of training optical NNs and provides insights into leveraging physical transformations for enhancing the NN performance.

Stain-free identification of cell nuclei using tomographic phase microscopy in flow cytometry.

Quantitative Phase Imaging (QPI) has gained popularity in bioimaging because it can avoid the need for cell staining, which in some cases is difficult or impossible. However, as a result, QPI does not provide labelling of various specific intracellular structures. Here we show a novel computational segmentation method based on statistical inference that makes it possible for QPI techniques to identify the cell nucleus. We demonstrate the approach with refractive index tomograms of stain-free cells reconstructed through the tomographic phase microscopy in flow cytometry mode. In particular, by means of numerical simulations and two cancer cell lines, we demonstrate that the nucleus can be accurately distinguished within the stain-free tomograms. We show that our experimental results are consistent with confocal fluorescence microscopy (FM) data and microfluidic cytofluorimeter outputs. This is a significant step towards extracting specific three-dimensional intracellular structures directly from the phase-contrast data in a typical flow cytometry configuration.

Photonic waveguide bundles using 3D laser writing and deep neural network image reconstruction.

In recent years, three-dimensional (3D) printing with multi-photon laser writing has become an essential tool for the manufacturing of three-dimensional optical elements. Single-mode optical waveguides are one of the fundamental photonic components, and are the building block for compact multicore fiber bundles, where thousands of single-mode elements are closely packed, acting as individual pixels and delivering the local information to a sensor. In this work, we present the fabrication of polymer rectangular step-index (STIN) optical waveguide bundles in the IP-Dip photoresist, using a commercial 3D printer. Moreover, we reduce the core-to-core spacing of the imaging bundles by means of a deep neural network (DNN) which has been trained with a large synthetic dataset, demonstrating that the scrambling of information due to diffraction and cross-talk between fiber cores can be undone. The DNN-based approach can be adopted in applications such as on-chip platforms and microfluidic systems where accurate imaging from in-situ printed fiber bundles suffer cross-talk. In this respect, we provide a design and fabrication guideline for such scenarios by employing the DNN not only as a post-processing technique but also as a design optimization tool.

High speed, complex wavefront shaping using the digital micro-mirror device.

Digital micro-mirror devices (DMDs) have been deployed in many optical applications. As compared to spatial light modulators (SLMs), they are characterized by their much faster refresh rates (full-frame refresh rates up to 32 kHz for binary patterns) compared to 120 Hz for most liquid crystal SLMs. DMDs however can only display binary, unipolar patterns and utilize temporal modulation to represent with excellent accuracy multiple gray-levels in display applications. We used the built-in time domain dynamic range representation of the DMD to project 8-bit complex-fields. With this method, we demonstrated 8-bit complex field modulation with a frame time of 38.4 ms (around 0.15 s for the entire complex-field). We performed phase conjugation by compensating the distortions incurred due to propagation through free-space and a scattering medium. For faster modulation speed, an electro-optic modulator was used in synchronization with the DMD in an amplitude modulation mode to create grayscale patterns with frame rate ~ 833 Hz with display time of only 1.2 ms instead of 38.4 ms for time multiplexing gaining a speed up by a factor of 32.

A membrane-less electrolyzer with porous walls for high throughput and pure hydrogen production.

Membrane-less electrolyzers utilize fluidic forces instead of solid barriers for the separation of electrolysis gas products. These electrolyzers have low ionic resistance, a simple design, and the ability to work with electrolytes at different pH values. However, the interelectrode distance and the flow velocity should be large at high production rates to prevent gas cross over. This is not energetically favorable as the ionic resistance is higher at larger interelectrode distances and the required pumping power increases with the flow velocity. In this work, a new solution is introduced to increase the throughput of electrolyzers without the need for increasing these two parameters. The new microfluidic reactor has three channels separated by porous walls. The electrolyte enters the middle channel and flows into the outer channels through the wall pores. Gas products are being produced in the outer channels. Hydrogen cross over is 0.14% in this electrolyzer at flow rate = 80 mL h-1 and current density (j) = 300 mA cm-2. This cross over is 58 times lower than hydrogen cross over in an equivalent membrane-less electrolyzer with parallel electrodes under the same working conditions. Moreover, the addition of a surfactant to the electrolyte further reduces the hydrogen cross over by 21% and the overpotential by 1.9%. This is due to the positive effects of surfactants on the detachment and coalescence dynamics of bubbles. The addition of the passive additive and implementation of the porous walls result in twice the hydrogen production rate in the new reactor compared to parallel electrode electrolyzers with similar hydrogen cross over.

3D reconstruction of weakly scattering objects from 2D intensity-only measurements using the Wolf transform.

A new approach to optical diffraction tomography (ODT) based on intensity measurements is presented. By applying the Wolf transform directly to intensity measurements, we observed unexpected behavior in the 3D reconstruction of the sample. Such a reconstruction does not explicitly represent a quantitative measure of the refractive index of the sample; however, it contains interesting qualitative information. This 3D reconstruction exhibits edge enhancement and contrast enhancement for nanostructures compared with the conventional 3D refractive index reconstruction and thus could be used to localize nanoparticles such as lipids inside a biological sample.

Fluorescence-Based and Fluorescent Label-Free Characterization of Polymer Nanoparticle Decorated T Cells.

Cells are attractive carriers for the transport and delivery of nanoparticulate cargo. The use of cell-based carriers allows one to enhance control over the biodistribution of drug-loaded polymers and polymer nanoparticles. One key element in the development of cell-based delivery systems is the loading of the cell-based carrier with the nanoparticle cargo, which can be achieved either by internalization of the payload or by immobilization on the cell surface. The surface modification of cells with nanoparticles or the internalization of nanoparticles by cells is usually monitored with fluorescence-based techniques, such as flow cytometry and confocal microscopy. In spite of the widespread use of these techniques, the use of fluorescent labels also poses some risks and has several drawbacks. Fluorescent dyes may bleach, or leach from, the nanoparticles or alter the physicochemical properties of nanoparticles and their interactions with and uptake by cells. Using poly(d,l-lactic acid) nanoparticles that are loaded with Coumarin 6, BODIPY 493/503, or DiO dyes as a model system, this paper demonstrates that the use of physically entrapped fluorescent labels can lead to false negative or erroneous results. The use of nanoparticles that contain covalently tethered fluorescent dyes instead was found to provide a robust approach to monitor cell surface conjugation reactions and to quantitatively analyze nanoparticle-decorated cells. Finally, it is shown that optical diffraction tomography is an attractive, alternative technique for the characterization of nanoparticle-decorated cells, which obviates the need for fluorescent labels.

Scalable optical learning operator.

Today's heavy machine learning tasks are fueled by large datasets. Computing is performed with power-hungry processors whose performance is ultimately limited by the data transfer to and from memory. Optics is a powerful means of communicating and processing information, and there is currently intense interest in optical information processing for realizing high-speed computations. Here we present and experimentally demonstrate an optical computing framework called scalable optical learning operator, which is based on spatiotemporal effects in multimode fibers for a range of learning tasks including classifying COVID-19 X-ray lung images, speech recognition and predicting age from images of faces. The presented framework addresses the energy scaling problem of existing systems without compromising speed. We leverage simultaneous, linear and nonlinear interaction of spatial modes as a computation engine. We numerically and experimentally show the ability of the method to execute several different tasks with accuracy comparable with a digital implementation.

Inference in artificial intelligence with deep optics and photonics.

Artificial intelligence tasks across numerous applications require accelerators for fast and low-power execution. Optical computing systems may be able to meet these domain-specific needs but, despite half a century of research, general-purpose optical computing systems have yet to mature into a practical technology. Artificial intelligence inference, however, especially for visual computing applications, may offer opportunities for inference based on optical and photonic systems. In this Perspective, we review recent work on optical computing for artificial intelligence applications and discuss its promise and challenges.

Fabrication of Sub-Micron Polymer Waveguides through Two-Photon Polymerization in Polydimethylsiloxane.

Flexible ultra-compact low-loss optical waveguides play a vital role in the development of soft photonics. The search for suitable materials and innovative fabrication techniques to achieve low loss long polymer optical waveguides and interconnects has proven to be challenging. In this paper, we demonstrate the fabrication of submicron optical waveguides in polydimethylsiloxane (PDMS) using divinylbenzene (DVB) as the photopolymerizable monomer through two-photon polymerization (2PP). We show that the commercial oxime ester photoinitiator Irgacure OXE02 is suitable for triggering the DVB photopolymerization, resulting in a stable and controllable fabrication process for the fabrication of defect-free, 5-cm long waveguides. We further explore a multi-track fabrication strategy to enlarge the waveguide core size up to ~3 µm for better light confinement and reduced cross-talk. In these waveguides, we measured a refractive index contrast on the order of 0.005 and a transmission loss of 0.1 dB/cm at 710 nm wavelength.

All-fiber spatiotemporally mode-locked laser with multimode fiber-based filtering.

We demonstrate the first all-fiber multimode spatiotemporally mode-locked laser. The oscillator generates dissipative soliton pulses at 1036 nm with 12 mW average power, 6.24 ps duration, and 24.3 MHz repetition rate. The reported pulse energy (0.5 nJ) represents ∼4 times improvement over the previously reported single-mode all-normal dispersion mode-locked lasers with multimode interference-based filtering. Numerical simulations are performed to investigate the cavity and spatiotemporal mode-locking dynamics. The all-fiber oscillator we present shows promise for practical use since it can be fabricated simply.

Microfluidic-based linear-optics label-free imager.

Linear optics based nanoscopy previously reached resolution beyond the diffraction limit, illuminating samples in the visible light regime while allowing light to interact with freely moving metallic nanoparticles. However, the hydrodynamics governing the nanoparticle motion used to scan the sample is very complex and has low probability of achieving appropriate and fast mapping in practice. Hence, an implementation of the technique on real biological samples has not been demonstrated so far. Moreover, a suitable way to perform controlled nanoparticle scanning of biological samples is required. Here we show a solution where a microfluidic channel is used to flow and trap biological samples inside a water droplet along with suspended nanoparticles surrounded by silicone oil. The evanescent light scattered from the sample and is rescattered by the nanoparticles in the vicinity. This encodes the sub-wavelength features of the sample which can later on be decoded and reconstructed from measurements in the far field. The microfluidic system-controlled flow allows better nanoparticle scanning of the sample and maintains an isolated system for each sample in each droplet. A more localized scan at the droplet water/oil interface is also conducted using amphiphilic nanoparticles where their hydrophilic side is constrained to the droplet and their hydrophobic side is constrained to the oil. This allows higher probability of capturing evanescent fields closer to their origin, yielding better resolution and a higher signal to noise ratio. Using this system, we obtained images of an E. coli sample and demonstrated how the method yield fine resolution of the sample contours. To the best of our knowledge, this is the first time that a linear and label free optics imaging process was performed using a micro-fluidic device.

In Vitro Cytocompatibility Assessment of Ti-Modified, Silicon-oxycarbide-Based, Polymer-Derived, Ceramic-Implantable Electrodes under Pacing Conditions.

Polymer-derived ceramics (PDC) have recently gained increased interest in the field of bioceramics. Among PDC's, carbon-rich silicon oxycarbide ceramics (SiOC) possess good combined electrical and mechanical properties. Their durability in aggressive environments and proposed cytocompatibility makes them an attractive material for fabrication of bio-MEMS devices such as pacemaker electrodes. The aim of the present study is to demonstrate the remarkable mechanical and electrical properties, biological response of PDCs modified with titanium (Ti) and their potential for application as pacemaker electrodes. Therefore, a new type of SiOC modified with Ti fillers was synthesized via PDC route using a Pt-catalyzed hydrosilylation reaction. Preceramic green bodies were pyrolyzed at 1000 °C under an argon atmosphere to achieve amorphous ceramics. Electrical and mechanical characterization of SiCxO2(1-x)/TiOxCy ceramics revealed a maximum electrical conductivity of 10 S cm-1 and a flexural strength of maximal 1 GPa, which is acceptable for pacemaker applications. Ti incorporation is found to be beneficial for enhancing the electrical conductivity of SiOC ceramics and the conductivity values were increased with Ti doping and reached a maximum for the composition with 30 wt % Ti precursor. Cytocompatibility was demonstrated for the PDC SiOC ceramics as well as SiOC ceramics modified with Ti fillers. Cytocompatibility was also demonstrated for SiTiOC20 electrodes under pacing conditions by monitoring of cells in an in vitro 3D environment. Collectively, these data demonstrate the great potential of polymer-derived SiOC ceramics to be used as pacemaker electrodes.

Imaging hair cells through laser-ablated cochlear bone.

We report an innovative technique for the visualization of cells through an overlying scattering medium by combining femtosecond laser bone ablation and two-photon excitation fluorescence (TPEF) microscopy. We demonstrate the technique by imaging hair cells in an intact mouse cochlea ex vivo. Intracochlear imaging is important for the assessment of hearing disorders. However, the small size of the cochlea and its encasement in the densest bone in the body present challenging obstacles, preventing the visualization of the intracochlear microanatomy using standard clinical imaging modalities. The controlled laser ablation reduces the optical scattering of the cochlear bone while the TPEF allows visualization of individual cells behind the bone. We implemented optical coherence tomography (OCT) simultaneously with the laser ablation to enhance the precision of the ablation and prevent inadvertent damage to the cells behind the bone.

High-fidelity optical diffraction tomography of multiple scattering samples.

We propose an iterative reconstruction scheme for optical diffraction tomography that exploits the split-step non-paraxial (SSNP) method as the forward model in a learning tomography scheme. Compared with the beam propagation method (BPM) previously used in learning tomography (LT-BPM), the improved accuracy of SSNP maximizes the information retrieved from measurements, relying less on prior assumptions about the sample. A rigorous evaluation of learning tomography based on SSNP (LT-SSNP) using both synthetic and experimental measurements confirms its superior performance compared with that of the LT-BPM. Benefiting from the accuracy of SSNP, LT-SSNP can clearly resolve structures that are highly distorted in the LT-BPM. A serious limitation for quantifying the reconstruction accuracy for biological samples is that the ground truth is unknown. To overcome this limitation, we describe a novel method that allows us to compare the performances of different reconstruction schemes by using the discrete dipole approximation to generate synthetic measurements. Finally, we explore the capacity of learning approaches to enable data compression by reducing the number of scanning angles, which is of particular interest in minimizing the measurement time.

Learning from droplet flows in microfluidic channels using deep neural networks.

A non-intrusive method is presented for measuring different fluidic properties in a microfluidic chip by optically monitoring the flow of droplets. A neural network is used to extract the desired information from the images of the droplets. We demonstrate the method in two applications: measurement of the concentration of each component of a water/alcohol mixture, and measurement of the flow rate of the same mixture. A large number of droplet images are recorded and used to train deep neural networks (DNN) to predict the flow rate or the concentration. It is shown that this method can be used to quantify the concentrations of each component with a 0.5% accuracy and the flow rate with a resolution of 0.05 ml/h. The proposed method can in principle be used to measure other properties of the fluid such as surface tension and viscosity.

Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy.

Deep neural networks have been used to map multi-modal, multi-photon microscopy measurements of a label-free tissue sample to its corresponding histologically stained brightfield microscope colour image. It is shown that the extra structural and functional contrasts provided by using two source modes, namely two-photon excitation microscopy and fluorescence lifetime imaging, result in a more faithful reconstruction of the target haematoxylin and eosin stained mode. This modal mapping procedure can aid histopathologists, since it provides access to unobserved imaging modalities, and translates the high-dimensional numerical data generated by multi-modal, multi-photon microscopy into traditionally accepted visual forms. Furthermore, by combining the strengths of traditional chemical staining and modern multi-photon microscopy techniques, modal mapping enables label-free, non-invasive studies of in vivo tissue samples or intravital microscopic imaging inside living animals. The results show that modal co-registration and the inclusion of spatial variations increase the visual accuracy of the mapped results.