Transform- and multi-domain deep learning for single-frame rapid autofocusing in whole slide imaging.

A whole slide imaging (WSI) system has recently been approved for primary diagnostic use in the US. The image quality and system throughput of WSI is largely determined by the autofocusing process. Traditional approaches acquire multiple images along the optical axis and maximize a figure of merit for autofocusing. Here we explore the use of deep convolution neural networks (CNNs) to predict the focal position of the acquired image without axial scanning. We investigate the autofocusing performance with three illumination settings: incoherent Kohler illumination, partially coherent illumination with two plane waves, and one-plane-wave illumination. We acquire ~130,000 images with different defocus distances as the training data set. Different defocus distances lead to different spatial features of the captured images. However, solely relying on the spatial information leads to a relatively bad performance of the autofocusing process. It is better to extract defocus features from transform domains of the acquired image. For incoherent illumination, the Fourier cutoff frequency is directly related to the defocus distance. Similarly, autocorrelation peaks are directly related to the defocus distance for two-plane-wave illumination. In our implementation, we use the spatial image, the Fourier spectrum, the autocorrelation of the spatial image, and combinations thereof as the inputs for the CNNs. We show that the information from the transform domains can improve the performance and robustness of the autofocusing process. The resulting focusing error is ~0.5 µm, which is within the 0.8-µm depth-of-field range. The reported approach requires little hardware modification for conventional WSI systems and the images can be captured on the fly without focus map surveying. It may find applications in WSI and time-lapse microscopy. The transform- and multi-domain approaches may also provide new insights for developing microscopy-related deep-learning networks. We have made our training and testing data set (~12 GB) open-source for the broad research community.

Rapid volumetric imaging with Bessel-Beam three-photon microscopy.

Owing to its tissue-penetration ability, multi-photon fluorescence microscopy allows for the high-resolution, non-invasive imaging of deep tissue in vivo; the recently developed three-photon microscopy (3PM) has extended the depth of high-resolution, non-invasive functional imaging of mouse brains to beyond 1.0 mm. However, the low repetition rate of femtosecond lasers that are normally used in 3PM limits the temporal resolution of point-scanning three-photon microscopy. To increase the volumetric imaging speed of 3PM, we propose a combination of an axially elongated needle-like Bessel-beam with three-photon excitation (3PE) to image biological samples with an extended depth of focus. We demonstrate the higher signal-to-background ratio (SBR) of the Bessel-beam 3PM compared to the two-photon version both theoretically and experimentally. Finally, we perform simultaneous calcium imaging of brain regions at different axial locations in live fruit flies and rapid volumetric imaging of neuronal structures in live mouse brains. These results highlight the unique advantage of conducting rapid volumetric imaging with a high SBR in the deep brain in vivo using scanning Bessel-3PM.

Compact, modular and in-plane AOSLO for high-resolution retinal imaging.

The adaptive optics scanning laser ophthalmoscope (AOSLO) was first developed in 2002 and since then the technology has been adopted in several laboratories around the world, for both clinical and psychophysical research. There have been a few major design implementations of the AOSLO. The first used on-axis tilted spherical mirrors in a planar arrangement, and the second minimized the build up of astigmatism present in the first design by using a non-planar arrangement. Other designs have avoided astigmatism by using custom-made toroidal mirrors or by using lenses on-axis, rather than mirrors. We present a new design implementation for an AOSLO that maintains a planar optical alignment without the build up astigmatism using compact, reconfigurable modules based on an Offner relay system. We additionally use an off-the-shelf digital oscilloscope for data capture and custom-written Python code for generating and analyzing the retinal images. This design results in a compact system that is simple to align and, being composed of modular relays, has the potential for additional components to be added. We show that this system maintains diffraction-limited image quality across the field of view and that cones are clearly resolved in the central retina. The modular relay design is generally applicable to any system requiring one or more components in the pupil conjugate plane. This is likely to be useful for any point-scanned system, such as a standard scanning laser ophthalmoscope or non-ophthalmic confocal imaging system.

Line-scanning hyperspectral imaging based on structured illumination optical sectioning.

Line-scanning hyperspectral imaging (LHSI) is known to have a higher acquisition rate but lower sectioning capability than point-scanning hyperspectral imaging. To further increase the axial imaging contrast of LHSI, structured illumination was integrated into line excitation to remove the off-focus and scattered on-focus fluorescence signals. In an unsectioned leaf, the imaging contrast can be enhanced by 8 times, while in sectioned mouse skin tissues, a 4.5-fold enhancement can be achieved. With a spectral resolution of 1.15 nm, the fluorophores with seriously-overlapped spectra was proved to be separated without cross-talk by applying linear unmixing to the recorded spectral information.

Imaging of neuronal tissues by x-ray diffraction and x-ray fluorescence microscopy: evaluation of contrast and biomarkers for neurodegenerative diseases.

We have used scanning X-ray diffraction (XRD) and X-ray fluorescence (XRF) with micro-focused synchrotron radiation to study histological sections from human substantia nigra (SN). Both XRF and XRD mappings visualize tissue properties, which are inaccessible by conventional microscopy and histology. We propose to use these advanced tools to characterize neuronal tissue in neurodegeneration, in particular in Parkinson's disease (PD). To this end, we take advantage of the recent experimental progress in x-ray focusing, detection, and use automated data analysis scripts to enable quantitative analysis of large field of views. XRD signals are recorded and analyzed both in the regime of small-angle (SAXS) and wide-angle x-ray scattering (WAXS). The SAXS signal was analyzed in view of the local myelin structure, while WAXS was used to identify crystalline deposits. PD tissue scans exhibited increased amounts of crystallized cholesterol. The XRF analysis showed increased amounts of iron and decreased amounts of copper in the PD tissue compared to the control.

Optimizing pulse compressibility in completely all-fibered Ytterbium chirped pulse amplifiers for in vivo two photon laser scanning microscopy.

A simple and completely all-fiber Yb chirped pulse amplifier that uses a dispersion matched fiber stretcher and a spliced-on hollow core photonic bandgap fiber compressor is applied in nonlinear optical microscopy. This stretching-compression approach improves compressibility and helps to maximize the fluorescence signal in two-photon laser scanning microscopy as compared with approaches that use standard single mode fibers as stretcher. We also show that in femtosecond all-fiber systems, compensation of higher order dispersion terms is relevant even for pulses with relatively narrow bandwidths for applications relying on nonlinear optical effects. The completely all-fiber system was applied to image green fluorescent beads, a stained lily-of-the-valley root and rat-tail tendon. We also demonstrated in vivo imaging in zebrafish larvae, where we simultaneously measure second harmonic and fluorescence from two-photon excited red-fluorescent protein. Since the pulses are compressed in a fiber, this source is especially suited for upgrading existing laser scanning (confocal) microscopes with multiphoton imaging capabilities in space restricted settings or for incorporation in endoscope-based microscopy.

Two-photon laser scanning microscopy with electrowetting-based prism scanning.

Laser scanners are an integral part of high resolution biomedical imaging systems such as confocal or 2-photon excitation (2PE) microscopes. In this work, we demonstrate the utility of electrowetting on dielectric (EWOD) prisms as a lateral laser-scanning element integrated in a conventional 2PE microscope. To the best of our knowledge, this is the first such demonstration for EWOD prisms. EWOD devices provide a transmissive, low power consuming, and compact alternative to conventional adaptive optics, and hence this technology has tremendous potential. We demonstrate 2PE microscope imaging of cultured mouse hippocampal neurons with a FOV of 130 × 130 µm2 using EWOD prism scanning. In addition, we show simulations of the optical system with the EWOD prism, to evaluate the effect of propagating a Gaussian beam through the EWOD prism on the imaging quality. Based on the simulation results a beam size of 0.91 mm full width half max was chosen to conduct the imaging experiments, resulting in a numerical aperture of 0.17 of the imaging system.

Multiphoton excited hemoglobin fluorescence and third harmonic generation for non-invasive microscopy of stored blood.

Red blood cells (RBC) in two-photon excited fluorescence (TPEF) microscopy usually appear as dark disks because of their low fluorescent signal. Here we use 15fs 800nm pulses for TPEF, 45fs 1060nm pulses for three-photon excited fluorescence, and third harmonic generation (THG) imaging. We find sufficient fluorescent signal that we attribute to hemoglobin fluorescence after comparing time and wavelength resolved spectra of other expected RBC endogenous fluorophores: NADH, FAD, biliverdin, and bilirubin. We find that both TPEF and THG microscopy can be used to examine erythrocyte morphology non-invasively without breaching a blood storage bag.

Dual-modality endomicroscopy with co-registered fluorescence and phase contrast.

We describe a dual-modality laser scanning endomicroscope that provides simultaneous fluorescence contrast based on confocal laser endomicroscopy (CLE) and phase-gradient contrast based on scanning oblique back-scattering microscopy (sOBM). The probe consists of a 2.6mm-diameter micro-objective attached to a 30,000-core flexible fiber bundle. The dual contrasts are inherently co-registered, providing complementary information on labeled and un-labeled sample structure. Proof of principle demonstrations are presented with ex-vivo mouse colon tissue.

Label-free multi-photon imaging of dysplasia in Barrett's esophagus.

Barrett's esophagus (BE) is a metaplastic disorder where dysplastic and early cancerous changes are invisible to the naked eye and where the practice of blind biopsy is hampered by large sampling errors. Multi-photon microscopy (MPM) has emerged as an alternative solution for fast and label-free diagnostic capability for identifying the histological features with sub-micron accuracy. We developed a compact, inexpensive MPM system by using a handheld mode-locked fiber laser operating at 1560nm to study mucosal biopsies of BE. The combination of back-scattered THG, back-reflected forward THG and SHG signals generate images of cell nuclei and collagen, leading to label-free diagnosis in Barrett's.

Pump-probe imaging of pigmented cutaneous melanoma primary lesions gives insight into metastatic potential.

Metastatic melanoma is associated with a poor prognosis, but no method reliably predicts which melanomas of a given stage will ultimately metastasize and which will not. While sentinel lymph node biopsy (SLNB) has emerged as the most powerful predictor of metastatic disease, the majority of people dying from metastatic melanoma still have a negative SLNB. Here we analyze pump-probe microscopy images of thin biopsy slides of primary melanomas to assess their metastatic potential. Pump-probe microscopy reveals detailed chemical information of melanin with subcellular spatial resolution. Quantification of the molecular signatures without reference standards is achieved using a geometrical representation of principal component analysis. Melanin structure is analyzed in unison with the chemical information by applying principles of mathematical morphology. Results show that melanin in metastatic primary lesions has lower chemical diversity than non-metastatic primary lesions, and contains two distinct phenotypes that are indicative of aggressive disease. Further, the mathematical morphology analysis reveals melanin in metastatic primary lesions has a distinct "dusty" quality. Finally, a statistical analysis shows that the combination of the chemical information with spatial structures predicts metastatic potential with much better sensitivity than SLNB and high specificity, suggesting pump-probe microscopy can be an important tool to help predict the metastatic potential of melanomas.

Real-time GPU-accelerated processing and volumetric display for wide-field laser-scanning optical-resolution photoacoustic microscopy.

Fast signal processing and real-time displays are essential for practical imaging modality in various fields of applications. However, the imaging speed in optical-resolution photoacoustic microscopy (OR-PAM), in particular, depends on factors such as the pulse repetition rate of the laser, scanning method, field of view (FOV), and signal processing time. In the past, efforts to increase acquisition speed either focused on developing new scanning methods or using lasers with higher pulse repetition rates. However, high-speed signal processing is also important for real-time volumetric display in OR-PAM. In this study, we carried out parallel signal processing using a graphics processing unit (GPU) to enable fast signal processing and wide-field real-time displays in laser-scanning OR-PAM. The average total GPU processing time for a B-mode PAM image was approximately 1.35 ms at a display speed of 480 fps when the data samples were acquired with 736 (axial) × 500 (lateral) points/B-mode-frame at a pulse repetition rate of 300 kHz. In addition, we successfully displayed maximum amplitude projection images of a mouse's ear as volumetric images with an FOV of 3 mm × 3 mm (500 × 500 pixels) at 1.02 s, corresponding to 0.98 fps.

Rapid imaging of large tissues using high-resolution stage-scanning microscopy.

Rapid and high-resolution imaging of large tissues is essential in biological research, like brain neuron connectivity research and cancer margins imaging. Here a novel stage-scanning confocal microscopy was developed for rapid imaging of large tissues. Line scanning methods and strip imaging strategy were used to increase the imaging speed. The scientific CMOS was used as line detector in sub-array mode and the optical sectioning ability can be easily adjusted by changing the number of line detectors according to different samples. Fluorescent beads imaging showed resolutions of 0.47 µm, 0.56 µm, and 1.56 µm in the X, Y, and Z directions, respectively, with a 40 × objective lens. A 10 × 10 mm(2) coronal plane with enough signal intensity could be imaged in about 88 sec at a sampling resolution of 0.16 µm/pixel. Rapid imaging of mouse brain slices demonstrated the applicability of this system in visualizing neuronal details at high frame rate.

Phase-gradient contrast in thick tissue with a scanning microscope.

It is well known that the principle of reciprocity is valid for light traveling even through scattering or absorptive media. This principle has been used to establish an equivalence between conventional widefield microscopes and scanning microscopes. We make use of this principle to introduce a scanning version of oblique back-illumination microscopy, or sOBM. This technique provides sub-surface phase-gradient and amplitude images from unlabeled tissue, in an epi-detection geometry. That is, it may be applied to arbitrarily thick tissue. sOBM may be implemented as a simple, cost-effective add-on with any scanning microscope, requiring only the availability of an extra input channel in the microscope electronics. We demonstrate here its implementation in combination with two-photon excited fluorescence (TPEF) microscopy and with coherent anti-Stokes Raman scattering (CARS) microscopy, applied to brain or spinal cord tissue imaging. In both cases, sOBM provides information on tissue morphology complementary to TPEF or CARS contrast. This information is obtained simultaneously and is automatically co-registered. Finally, we show that sOBM can be operated at video rate.

Characterization of multiphoton microscopy in the bone marrow following intravital laser osteotomy.

The bone marrow is an important site where all blood cells are formed from hematopoietic stem cells and where hematologic malignancies such as leukemia emerge. It is also a frequent site for metastasis of solid tumors such as breast cancer and prostate cancer. Intravital microscopy is a powerful tool for studying the bone marrow with single cell and sub-cellular resolution. To improve optical access to this rich biological environment, plasma-mediated laser ablation with sub-microjoule femtosecond pulses was used to thin cortical bone. By locally removing a superficial layer of bone (local laser osteotomy), significant improvements in multiphoton imaging were observed in individual bone marrow compartments in vivo. This work demonstrates the utility of scanning laser ablation of hard tissue with sub-microjoule pulses as a preparatory step to imaging.

In vivo micro-vascular imaging and flow cytometry in zebrafish using two-photon excited endogenous fluorescence.

Zebrafish has rapidly evolved as a powerful vertebrate model organism for studying human diseases. Here we first demonstrate a new label-free approach for in vivo imaging of microvasculature, based on the recent discovery and detailed characterization of the two-photon excited endogenous fluorescence in the blood plasma of zebrafish. In particular, three-dimensional reconstruction of the microvascular networks was achieved with the depth-resolved two-photon excitation fluorescence (TPEF) imaging. Secondly, the blood flow images, obtained by perpendicularly scanning the focal point across the blood vessel, provided accurate information for characterizing the hemodynamics of the circulatory system. The endogenous fluorescent signals of reduced nicotinamide adenine dinucleotide (NADH) enabled visualization of the circulating granulocytes (neutrophils) in the blood vessel. The development of acute sterile inflammation could be detected by the quantitative counting of circulating neutrophils. Finally, we found that by utilizing a short wavelength excitation at 650 nm, the commonly used fluorescent proteins, such as GFP and DsRed, could be efficiently excited together with the endogenous fluorophores to achieve four-color TPEF imaging of the vascular structures and blood cells. The results demonstrated that the multi-color imaging could potentially yield multiple view angles of important processes in living biological systems.

Non-de-scanned parallel recording two-photon hyperspectral microscopy with high spectral and spatial resolution.

A two-photon hyperspectral microscope with non-de-scanned geometry and parallel recording scheme was constructed which had a low dwell time, high spectral resolution, and high spatial resolution. The dwell time was effectively decreased to reduce photodamage through parallel recording, while the non-de-scanned geometry led to a considerable reduction in the signal loss and spectrum distortion. Two experiments were performed to show that this system can solve crosstalk issues and spectrally resolve the intrinsic fluorophores in optically-thick tissues without staining and sectioning.

Dynamic autofluorescence imaging of intracellular components inside living cells using direct electron beam excitation.

We developed a high-resolution fluorescence microscope in which fluorescent materials are directly excited using a focused electron beam. Electron beam excitation enables detailed observations on the nanometer scale. Real-time live-cell observation is also possible using a thin film to separate the environment under study from the vacuum region required for electron beam propagation. In this study, we demonstrated observation of cellular components by autofluorescence excited with a focused electron beam and performed dynamic observations of intracellular granules. Since autofluorescence is associated with endogenous substances in cells, this microscope can also be used to investigate the intrinsic properties of organelles.

In-Vivo functional optical-resolution photoacoustic microscopy with stimulated Raman scattering fiber-laser source.

In this paper a multi-wavelength optical-resolution photoacoustic microscopy (OR-PAM) system using stimulated Raman scattering is demonstrated for both phantom and in vivo imaging. A 1-ns pulse width ytterbium-doped fiber laser is coupled into a single-mode polarization maintaining fiber. Discrete Raman-shifted wavelength peaks extending to nearly 800 nm are generated with pulse energies sufficient for OR-PAM imaging. Bandpass filters are used to select imaging wavelengths. A dual-mirror galvanometer system was used to scan the focused outputs across samples of carbon fiber networks, 200µm dye-filled tubes, and Swiss Webster mouse ears. Photoacoustic signals were collected in transmission mode and used to create maximum amplitude projection C-scan images. Double dye experiments and in vivo oxygen saturation estimation confirmed functional imaging potential.

Super-resolution scanning laser microscopy through virtually structured detection.

High resolution microscopy is essential for advanced study of biological structures and accurate diagnosis of medical diseases. The spatial resolution of conventional microscopes is light diffraction limited. Structured illumination has been extensively explored to break the diffraction limit in wide field light microscopy. However, deployable application of the structured illumination in scanning laser microscopy is challenging due to the complexity of the illumination system and possible phase errors in sequential illumination patterns required for super-resolution reconstruction. We report here a super-resolution scanning laser imaging system which employs virtually structured detection (VSD) to break the diffraction limit. Without the complexity of structured illumination, VSD provides an easy, low-cost and phase-artifact free strategy to achieve super-resolution in scanning laser microscopy.