Pan-cortical cellular imaging in freely behaving mice using a miniaturized micro-camera array microscope (mini-MCAM).

Understanding how circuits in the brain simultaneously coordinate their activity to mediate complex ethnologically relevant behaviors requires recording neural activities from distributed populations of neurons in freely behaving animals. Current miniaturized imaging microscopes are typically limited to imaging a relatively small field of view, precluding the measurement of neural activities across multiple brain regions. Here we present a miniaturized micro-camera array microscope (mini-MCAM) that consists of four fluorescence imaging micro-cameras, each capable of capturing neural activity across a 4.5 mm x 2.55 mm field of view (FOV). Cumulatively, the mini-MCAM images over 30 mm 2 area of sparsely expressed GCaMP6s neurons distributed throughout the dorsal cortex, in regions including the primary and secondary motor, somatosensory, visual, retrosplenial, and association cortices across both hemispheres. We demonstrate cortex-wide cellular resolution in vivo Calcium (Ca 2+ ) imaging using the mini-MCAM in both head-fixed and freely behaving mice.

Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second.

Wide field of view microscopy that can resolve 3D information at high speed and spatial resolution is highly desirable for studying the behaviour of freely moving model organisms. However, it is challenging to design an optical instrument that optimises all these properties simultaneously. Existing techniques typically require the acquisition of sequential image snapshots to observe large areas or measure 3D information, thus compromising on speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over an area of 135 cm2, achieving up to 230 frames per second at spatiotemporal throughputs exceeding 5 gigapixels per second. 3D-RAPID employs a 3D reconstruction algorithm that, for each synchronized snapshot, fuses all 54 images into a composite that includes a co-registered 3D height map. The self-supervised 3D reconstruction algorithm trains a neural network to map raw photometric images to 3D topography using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. The resulting reconstruction process is thus robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. We demonstrate the broad applicability of 3D-RAPID with collections of several freely behaving organisms, including ants, fruit flies, and zebrafish larvae.

scatterBrains: an open database of human head models and companion optode locations for realistic Monte Carlo photon simulations.

Significance: Monte Carlo (MC) simulations are currently the gold standard in the near-infrared and diffuse correlation spectroscopy (NIRS/DCS) communities for generating light transport paths through tissue. However, realistic and diverse models that capture complex tissue layers are not easily available to all; moreover, manually placing optodes on such models can be tedious and time consuming. Such limitations may hinder the adoption of representative models for basic simulations and the use of these models for large-scale simulations, e.g., for training machine learning algorithms. Aim: We aim to provide the NIRS/DCS communities with an open-source, user-friendly database of morphologically and optically realistic head models, as well as a succinct software pipeline to prepare these models for mesh-based Monte Carlo simulations of light transport. Approach: Sixteen anatomical models were created from segmented T1-weighted magnetic resonance imaging (MRI) head scans and converted to tetrahedral mesh volumes. Approximately 800 companion scalp surface locations were distributed on each model, comprising full head coverage. A pipeline was created to place custom source and optical detectors at each location, and guidance is provided on how to use these parameters to set up MC simulations. Results: The models, head surface locations, and all associated code are freely available under the scatterBrains project on Github. Conclusions: The NIRS/DCS community benefits from having shared resources for conducting MC simulations on realistic head geometries. We hope this will make MRI-based head models and virtual optode placement easily accessible to all. Contributions to the database are welcome and encouraged.

High-resolution single-photon imaging with physics-informed deep learning.

High-resolution single-photon imaging remains a big challenge due to the complex hardware manufacturing craft and noise disturbances. Here, we introduce deep learning into SPAD, enabling super-resolution single-photon imaging with enhancement of bit depth and imaging quality. We first studied the complex photon flow model of SPAD electronics to accurately characterize multiple physical noise sources, and collected a real SPAD image dataset (64 × 32 pixels, 90 scenes, 10 different bit depths, 3 different illumination flux, 2790 images in total) to calibrate noise model parameters. With this physical noise model, we synthesized a large-scale realistic single-photon image dataset (image pairs of 5 different resolutions with maximum megapixels, 17250 scenes, 10 different bit depths, 3 different illumination flux, 2.6 million images in total) for subsequent network training. To tackle the severe super-resolution challenge of SPAD inputs with low bit depth, low resolution, and heavy noise, we further built a deep transformer network with a content-adaptive self-attention mechanism and gated fusion modules, which can dig global contextual features to remove multi-source noise and extract full-frequency details. We applied the technique in a series of experiments including microfluidic inspection, Fourier ptychography, and high-speed imaging. The experiments validate the technique's state-of-the-art super-resolution SPAD imaging performance.

SEMPAI: a Self-Enhancing Multi-Photon Artificial Intelligence for Prior-Informed Assessment of Muscle Function and Pathology.

Deep learning (DL) shows notable success in biomedical studies. However, most DL algorithms work as black boxes, exclude biomedical experts, and need extensive data. This is especially problematic for fundamental research in the laboratory, where often only small and sparse data are available and the objective is knowledge discovery rather than automation. Furthermore, basic research is usually hypothesis-driven and extensive prior knowledge (priors) exists. To address this, the Self-Enhancing Multi-Photon Artificial Intelligence (SEMPAI) that is designed for multiphoton microscopy (MPM)-based laboratory research is presented. It utilizes meta-learning to optimize prior (and hypothesis) integration, data representation, and neural network architecture simultaneously. By this, the method allows hypothesis testing with DL and provides interpretable feedback about the origin of biological information in 3D images. SEMPAI performs multi-task learning of several related tasks to enable prediction for small datasets. SEMPAI is applied on an extensive MPM database of single muscle fibers from a decade of experiments, resulting in the largest joint analysis of pathologies and function for single muscle fibers to date. It outperforms state-of-the-art biomarkers in six of seven prediction tasks, including those with scarce data. SEMPAI's DL models with integrated priors are superior to those without priors and to prior-only approaches.

Multi-modal imaging using a cascaded microscope design.

We present a multi-modal fiber array snapshot technique (M-FAST) based on an array of 96 compact cameras placed behind a primary objective lens and a fiber bundle array. Our technique is capable of large-area, high-resolution, multi-channel video acquisition. The proposed design provides two key improvements to prior cascaded imaging system approaches: a novel optical arrangement that accommodates the use of planar camera arrays, and a new ability to acquire multi-modal image data acquisition. M-FAST is a multi-modal, scalable imaging system that can acquire snapshot dual-channel fluorescence images as well as differential phase contrast measurements over a large 6.59 mm × 9.74 mm field-of-view at 2.2-µm center full-pitch resolution.

Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second.

To study the behavior of freely moving model organisms such as zebrafish (Danio rerio) and fruit flies (Drosophila) across multiple spatial scales, it would be ideal to use a light microscope that can resolve 3D information over a wide field of view (FOV) at high speed and high spatial resolution. However, it is challenging to design an optical instrument to achieve all of these properties simultaneously. Existing techniques for large-FOV microscopic imaging and for 3D image measurement typically require many sequential image snapshots, thus compromising speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over a 135-cm^2 area, achieving up to 230 frames per second at throughputs exceeding 5 gigapixels (GPs) per second. 3D-RAPID features a 3D reconstruction algorithm that, for each synchronized temporal snapshot, simultaneously fuses all 54 images seamlessly into a globally-consistent composite that includes a coregistered 3D height map. The self-supervised 3D reconstruction algorithm itself trains a spatiotemporally-compressed convolutional neural network (CNN) that maps raw photometric images to 3D topography, using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. As a result, our end-to-end 3D reconstruction algorithm is robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. The scalable hardware and software design of 3D-RAPID addresses a longstanding problem in the field of behavioral imaging, enabling parallelized 3D observation of large collections of freely moving organisms at high spatiotemporal throughputs, which we demonstrate in ants (Pogonomyrmex barbatus), fruit flies, and zebrafish larvae.

Gigapixel imaging with a novel multi-camera array microscope.

The dynamics of living organisms are organized across many spatial scales. However, current cost-effective imaging systems can measure only a subset of these scales at once. We have created a scalable multi-camera array microscope (MCAM) that enables comprehensive high-resolution recording from multiple spatial scales simultaneously, ranging from structures that approach the cellular scale to large-group behavioral dynamics. By collecting data from up to 96 cameras, we computationally generate gigapixel-scale images and movies with a field of view over hundreds of square centimeters at an optical resolution of 18 µm. This allows us to observe the behavior and fine anatomical features of numerous freely moving model organisms on multiple spatial scales, including larval zebrafish, fruit flies, nematodes, carpenter ants, and slime mold. Further, the MCAM architecture allows stereoscopic tracking of the z-position of organisms using the overlapping field of view from adjacent cameras. Overall, by removing the bottlenecks imposed by single-camera image acquisition systems, the MCAM provides a powerful platform for investigating detailed biological features and behavioral processes of small model organisms across a wide range of spatial scales.

Optical imaging and spectroscopy for the study of the human brain: status report.

This report is the second part of a comprehensive two-part series aimed at reviewing an extensive and diverse toolkit of novel methods to explore brain health and function. While the first report focused on neurophotonic tools mostly applicable to animal studies, here, we highlight optical spectroscopy and imaging methods relevant to noninvasive human brain studies. We outline current state-of-the-art technologies and software advances, explore the most recent impact of these technologies on neuroscience and clinical applications, identify the areas where innovation is needed, and provide an outlook for the future directions.

Transient Motion Classification Through Turbid Volumes via Parallelized Single-Photon Detection and Deep Contrastive Embedding.

Fast noninvasive probing of spatially varying decorrelating events, such as cerebral blood flow beneath the human skull, is an essential task in various scientific and clinical settings. One of the primary optical techniques used is diffuse correlation spectroscopy (DCS), whose classical implementation uses a single or few single-photon detectors, resulting in poor spatial localization accuracy and relatively low temporal resolution. Here, we propose a technique termed C lassifying R apid decorrelation E vents via P arallelized single photon d E tection (CREPE), a new form of DCS that can probe and classify different decorrelating movements hidden underneath turbid volume with high sensitivity using parallelized speckle detection from a 32 × 32 pixel SPAD array. We evaluate our setup by classifying different spatiotemporal-decorrelating patterns hidden beneath a 5 mm tissue-like phantom made with rapidly decorrelating dynamic scattering media. Twelve multi-mode fibers are used to collect scattered light from different positions on the surface of the tissue phantom. To validate our setup, we generate perturbed decorrelation patterns by both a digital micromirror device (DMD) modulated at multi-kilo-hertz rates, as well as a vessel phantom containing flowing fluid. Along with a deep contrastive learning algorithm that outperforms classic unsupervised learning methods, we demonstrate our approach can accurately detect and classify different transient decorrelation events (happening in 0.1-0.4 s) underneath turbid scattering media, without any data labeling. This has the potential to be applied to non-invasively monitor deep tissue motion patterns, for example identifying normal or abnormal cerebral blood flow events, at multi-Hertz rates within a compact and static detection probe.

Imaging Dynamics Beneath Turbid Media via Parallelized Single-Photon Detection.

Noninvasive optical imaging through dynamic scattering media has numerous important biomedical applications but still remains a challenging task. While standard diffuse imaging methods measure optical absorption or fluorescent emission, it is also well-established that the temporal correlation of scattered coherent light diffuses through tissue much like optical intensity. Few works to date, however, have aimed to experimentally measure and process such temporal correlation data to demonstrate deep-tissue video reconstruction of decorrelation dynamics. In this work, a single-photon avalanche diode array camera is utilized to simultaneously monitor the temporal dynamics of speckle fluctuations at the single-photon level from 12 different phantom tissue surface locations delivered via a customized fiber bundle array. Then a deep neural network is applied to convert the acquired single-photon measurements into video of scattering dynamics beneath rapidly decorrelating tissue phantoms. The ability to reconstruct images of transient (0.1-0.4 s) dynamic events occurring up to 8 mm beneath a decorrelating tissue phantom with millimeter-scale resolution is demonstrated, and it is highlighted how the model can flexibly extend to monitor flow speed within buried phantom vessels.

Quantitative Jones matrix imaging using vectorial Fourier ptychography.

This paper presents a microscopic imaging technique that uses variable-angle illumination to recover the complex polarimetric properties of a specimen at high resolution and over a large field-of-view. The approach extends Fourier ptychography, which is a synthetic aperture-based imaging approach to improve resolution with phaseless measurements, to additionally account for the vectorial nature of light. After images are acquired using a standard microscope outfitted with an LED illumination array and two polarizers, our vectorial Fourier ptychography (vFP) algorithm solves for the complex 2x2 Jones matrix of the anisotropic specimen of interest at each resolved spatial location. We introduce a new sequential Gauss-Newton-based solver that additionally jointly estimates and removes polarization-dependent imaging system aberrations. We demonstrate effective vFP performance by generating large-area (29 mm2), high-resolution (1.24 µm full-pitch) reconstructions of sample absorption, phase, orientation, diattenuation, and retardance for a variety of calibration samples and biological specimens.

Increasing a microscope's effective field of view via overlapped imaging and machine learning.

This work demonstrates a multi-lens microscopic imaging system that overlaps multiple independent fields of view on a single sensor for high-efficiency automated specimen analysis. Automatic detection, classification and counting of various morphological features of interest is now a crucial component of both biomedical research and disease diagnosis. While convolutional neural networks (CNNs) have dramatically improved the accuracy of counting cells and sub-cellular features from acquired digital image data, the overall throughput is still typically hindered by the limited space-bandwidth product (SBP) of conventional microscopes. Here, we show both in simulation and experiment that overlapped imaging and co-designed analysis software can achieve accurate detection of diagnostically-relevant features for several applications, including counting of white blood cells and the malaria parasite, leading to multi-fold increase in detection and processing throughput with minimal reduction in accuracy.

A multiple instance learning approach for detecting COVID-19 in peripheral blood smears.

A wide variety of diseases are commonly diagnosed via the visual examination of cell morphology within a peripheral blood smear. For certain diseases, such as COVID-19, morphological impact across the multitude of blood cell types is still poorly understood. In this paper, we present a multiple instance learning-based approach to aggregate high-resolution morphological information across many blood cells and cell types to automatically diagnose disease at a per-patient level. We integrated image and diagnostic information from across 236 patients to demonstrate not only that there is a significant link between blood and a patient's COVID-19 infection status, but also that novel machine learning approaches offer a powerful and scalable means to analyze peripheral blood smears. Our results both backup and enhance hematological findings relating blood cell morphology to COVID-19, and offer a high diagnostic efficacy; with a 79% accuracy and a ROC-AUC of 0.90.

The Role of Machine Learning in Cardiovascular Pathology.

Machine learning has seen slow but steady uptake in diagnostic pathology over the past decade to assess digital whole-slide images. Machine learning tools have incredible potential to standardise, and likely even improve, histopathologic diagnoses, but they are not yet widely used in clinical practice. We describe the principles of these tools and technologies and some successful preclinical and pretranslational efforts in cardiovascular pathology, as well as a roadmap for moving forward. In nonhuman animal models, one proof-of-principle application is in rodent progressive cardiomyopathy, which is of particular significance to drug toxicity studies. Basic science successes include screening the quality of differentiated stem cells and characterising cardiomyocyte developmental stages, with potential applications for research and toxicology/drug safety screening using derived or native human pluripotent stem cells differentiated into cardiomyocytes. Translational studies of particular note include those with success in diagnosing the various forms of heart allograft rejection. For fully realising the value of these tools in clinical cardiovascular pathology, we identify 3 essential challenges. First is image quality standardisation to ensure that algorithms can be developed and implemented on robust, consistent data. The second is consensus diagnosis; experts don't always agree, and thus "truth" may be difficult to establish, but the algorithms themselves may provide a solution. The third is the need for large-enough data sets to facilitate robust algorithm development, necessitating large cross-institutional shared image databases. The power of histopathology-based machine learning technologies is tremendous, and we outline the next steps needed to capitalise on this power.

Ultrasensitive point-of-care immunoassay for secreted glycoprotein detects Ebola infection earlier than PCR.

Ebola virus (EBOV) hemorrhagic fever outbreaks have been challenging to deter due to the lack of health care infrastructure in disease-endemic countries and a corresponding inability to diagnose and contain the disease at an early stage. EBOV vaccines and therapies have improved disease outcomes, but the advent of an affordable, easily accessed, mass-produced rapid diagnostic test (RDT) that matches the performance of more resource-intensive polymerase chain reaction (PCR) assays would be invaluable in containing future outbreaks. Here, we developed and demonstrated the performance of a new ultrasensitive point-of-care immunoassay, the EBOV D4 assay, which targets the secreted glycoprotein of EBOV. The EBOV D4 assay is 1000-fold more sensitive than the U.S. Food and Drug Administration-approved RDTs and detected EBOV infection earlier than PCR in a standard nonhuman primate model. The EBOV D4 assay is suitable for low-resource settings and may facilitate earlier detection, containment, and treatment during outbreaks of the disease.

Deep image prior for undersampling high-speed photoacoustic microscopy.

Photoacoustic microscopy (PAM) is an emerging imaging method combining light and sound. However, limited by the laser's repetition rate, state-of-the-art high-speed PAM technology often sacrifices spatial sampling density (i.e., undersampling) for increased imaging speed over a large field-of-view. Deep learning (DL) methods have recently been used to improve sparsely sampled PAM images; however, these methods often require time-consuming pre-training and large training dataset with ground truth. Here, we propose the use of deep image prior (DIP) to improve the image quality of undersampled PAM images. Unlike other DL approaches, DIP requires neither pre-training nor fully-sampled ground truth, enabling its flexible and fast implementation on various imaging targets. Our results have demonstrated substantial improvement in PAM images with as few as 1.4 % of the fully sampled pixels on high-speed PAM. Our approach outperforms interpolation, is competitive with pre-trained supervised DL method, and is readily translated to other high-speed, undersampling imaging modalities.

Reconstructing Undersampled Photoacoustic Microscopy Images Using Deep Learning.

One primary technical challenge in photoacoustic microscopy (PAM) is the necessary compromise between spatial resolution and imaging speed. In this study, we propose a novel application of deep learning principles to reconstruct undersampled PAM images and transcend the trade-off between spatial resolution and imaging speed. We compared various convolutional neural network (CNN) architectures, and selected a Fully Dense U-net (FD U-net) model that produced the best results. To mimic various undersampling conditions in practice, we artificially downsampled fully-sampled PAM images of mouse brain vasculature at different ratios. This allowed us to not only definitively establish the ground truth, but also train and test our deep learning model at various imaging conditions. Our results and numerical analysis have collectively demonstrated the robust performance of our model to reconstruct PAM images with as few as 2% of the original pixels, which can effectively shorten the imaging time without substantially sacrificing the image quality.

Multi-element microscope optimization by a learned sensing network with composite physical layers.

Standard microscopes offer a variety of settings to help improve the visibility of different specimens to the end microscope user. Increasingly, however, digital microscopes are used to capture images for automated interpretation by computer algorithms (e.g., for feature classification, detection, or segmentation), often without any human involvement. In this work, we investigate an approach to jointly optimize multiple microscope settings, together with a classification network, for improved performance with such automated tasks. We explore the interplay between optimization of programmable illumination and pupil transmission, using experimentally imaged blood smears for automated malaria parasite detection, to show that multi-element "learned sensing" outperforms its single-element counterpart. While not necessarily ideal for human interpretation, the network's resulting low-resolution microscope images (20X-comparable) offer a machine learning network sufficient contrast to match the classification performance of corresponding high-resolution imagery (100X-comparable), pointing a path toward accurate automation over large fields-of-view.

Diffraction tomography with a deep image prior.

We present a tomographic imaging technique, termed Deep Prior Diffraction Tomography (DP-DT), to reconstruct the 3D refractive index (RI) of thick biological samples at high resolution from a sequence of low-resolution images collected under angularly varying illumination. DP-DT processes the multi-angle data using a phase retrieval algorithm that is extended by a deep image prior (DIP), which reparameterizes the 3D sample reconstruction with an untrained, deep generative 3D convolutional neural network (CNN). We show that DP-DT effectively addresses the missing cone problem, which otherwise degrades the resolution and quality of standard 3D reconstruction algorithms. As DP-DT does not require pre-captured data or pre-training, it is not biased towards any particular dataset. Hence, it is a general technique that can be applied to a wide variety of 3D samples, including scenarios in which large datasets for supervised training would be infeasible or expensive. We applied DP-DT to obtain 3D RI maps of bead phantoms and complex biological specimens, both in simulation and experiment, and show that DP-DT produces higher-quality results than standard regularization techniques. We further demonstrate the generality of DP-DT, using two different scattering models, the first Born and multi-slice models. Our results point to the potential benefits of DP-DT for other 3D imaging modalities, including X-ray computed tomography, magnetic resonance imaging, and electron microscopy.