On-the-fly compressive single-pixel foveation using the STOne transform.

Compressive imaging allows one to sample an image below the Nyquist rate yet still accurately recover it from the measurements by solving an L1 optimization problem. The L1 solvers, however, are iterative and can require significant time to reconstruct the original signal. Intuitively, the reconstruction time can be reduced by reconstructing fewer total pixels. The human eye reduces the total amount of data it processes by having a spatially varying resolution, a method called foveation. In this work, we use foveation to achieve a 4x improvement in L1 compressive sensing reconstruction speed for hyperspectral images and video. Unlike previous works, the presented technique allows the high-resolution region to be placed anywhere in the scene after the subsampled measurements have been acquired, has no moving parts, and is entirely non-adaptive.

Infrared object classification with a hybrid optical convolution neural network.

Recent advancements in machine vision have enabled a great range of applications from image classification to autonomous driving. However, there is still a dilemma between the pursuit of higher-resolution training images that require a detector array with more pixels on the front end, and the demands on acquisition for embedded systems restrained by power, transmission bandwidth, and storage. In this paper, a multi-pixel hybrid optical convolutional neural network machine vision system was designed and validated to perform high-speed infrared object detection. The proposed system replicates the front convolution layer in a convolutional neural network utilizing a high-speed digital micro-mirror device to display the first layer of kernels at a resolution greater than the subsequent detector. After this, further convolutions are carried out in software to perform the object recognition. An infrared vehicle dataset was used to validate the performance of the hybrid system through simulation. We also tested this in hardware by performing infrared classification on toy vehicles to showcase the feasibility of such a design.

General neural network approach to compressive feature extraction.

Computer vision with a single-pixel camera is currently limited by a trade-off between reconstruction capability and image classification accuracy. If random projections are used to sample the scene, then reconstruction is possible but classification accuracy suffers, especially in cases with significant background signal. If data-driven projections are used, then classification accuracy improves and the effect of the background is diminished, but image recovery is not possible. Here, we employ a shallow neural network to nonlinearly convert from measurements acquired with random patterns to measurements acquired with data-driven patterns. The results demonstrate that this improves classification accuracy while still allowing for full reconstruction.

A hyperspectral projector for simultaneous 3D spatial and hyperspectral imaging via structured illumination.

Both 3D imaging and hyperspectral imaging provide important information of the scene and combining them is beneficial in helping us perceive and understand real-world structures. Previous hyperspectral 3D imaging systems typically require a hyperspectral imaging system as the detector suffers from complicated hardware design, high cost, and high acquisition and reconstruction time. Here, we report a low-cost, high-frame rate, simple-design, and compact hyperspectral stripe projector (HSP) system based on a single digital micro-mirror device, capable of producing hyperspectral patterns where each row of pixels has an independently programmable spectrum. We demonstrate two example applications using the HSP via hyperspectral structured illumination: hyperspectral 3D surface imaging and spectrum-dependent hyperspectral compressive imaging of volume density of participating medium. The hyperspectral patterns simultaneously encode the 3D spatial and spectral information of the target, requiring only a grayscale sensor as the detector. The reported HSP and its applications provide a solution for combining structured illumination techniques with hyperspectral imaging in a simple, efficient, and low-cost manner. The work presented here represents a novel structured illumination technique that provides the basis and inspiration of future variations of hardware systems and software encoding schemes.

Spectroscopic imaging of surfaces-Sum frequency generation microscopy (SFGM) combined with compressive sensing (CS) technique.

Surface chemistry is notoriously difficult to study, in part, due to the decreased number of molecules that contribute to the properties compared to the bulk phase but often has significant effects on the chemical activity of the material. This is especially true in topics such as corrosion, catalysis, wetting, and many others in nature and industry. Sum frequency generation (SFG) spectroscopy was developed for interface studies due to its high molecular selectivity and surface sensitivity, which is quite useful to study the effects of structural inhomogeneity in microscopy. Compressive sensing (CS) combined with SFG spectroscopy minimizes the imaging time while still producing quality images. Selected systems are presented here to demonstrate the capability of CS-SFG microscopy. CS-SFG microscopy successfully distinguished the static monolayer molecular mixtures, the orientations and adsorption of adsorbed molecules by the dip-coating technique, and the localized CO behaviors on polycrystalline Pt electrodes. Further discussion includes dynamic imaging as a future direction in CS-SFG microscopy. As materials and surfaces become more complex, imaging with chemical contrast becomes indispensable to understanding their performance and CS-SFG microscopy seems highly beneficial in this respect.

Realization of hybrid compressive imaging strategies.

The tendency of natural scenes to cluster around low frequencies is not only useful in image compression, it also can prove advantageous in novel infrared and hyperspectral image acquisition. In this paper, we exploit this signal model with two approaches to enhance the quality of compressive imaging as implemented in a single-pixel compressive camera and compare these results against purely random acquisition. We combine projection patterns that can efficiently extract the model-based information with subsequent random projections to form the hybrid pattern sets. With the first approach, we generate low-frequency patterns via a direct transform. As an alternative, we also used principal component analysis of an image library to identify the low-frequency components. We present the first (to the best of our knowledge) experimental validation of this hybrid signal model on real data. For both methods, we acquire comparable quality of reconstructions while acquiring only half the number of measurements needed by traditional random sequences. The optimal combination of hybrid patterns and the effects of noise on image reconstruction are also discussed.

A compressive hyperspectral video imaging system using a single-pixel detector.

Capturing fine spatial, spectral, and temporal information of the scene is highly desirable in many applications. However, recording data of such high dimensionality requires significant transmission bandwidth. Current computational imaging methods can partially address this challenge but are still limited in reducing input data throughput. In this paper, we report a video-rate hyperspectral imager based on a single-pixel photodetector which can achieve high-throughput hyperspectral video recording at a low bandwidth. We leverage the insight that 4-dimensional (4D) hyperspectral videos are considerably more compressible than 2D grayscale images. We propose a joint spatial-spectral capturing scheme encoding the scene into highly compressed measurements and obtaining temporal correlation at the same time. Furthermore, we propose a reconstruction method relying on a signal sparsity model in 4D space and a deep learning reconstruction approach greatly accelerating reconstruction. We demonstrate reconstruction of 128 × 128 hyperspectral images with 64 spectral bands at more than 4 frames per second offering a 900× data throughput compared to conventional imaging, which we believe is a first-of-its kind of a single-pixel-based hyperspectral imager.

Kaiso regulates Znf131-mediated transcriptional activation.

Kaiso is a dual-specificity POZ-ZF transcription factor that regulates gene expression by binding to sequence-specific Kaiso binding sites (KBS) or methyl-CpG dinucleotide pairs. Kaiso was first identified as a binding partner for the epithelial cell adhesion regulator p120(ctn). The p120(ctn)/Kaiso interaction is reminiscent of the beta-catenin/TCF interaction and several studies have suggested that Kaiso is a negative regulator of the Wnt/beta-catenin TCF signaling pathway. To gain further insight into Kaiso's function, we performed a yeast two-hybrid screen using the Kaiso POZ domain as bait. This screen identified the POZ-ZF protein, Znf131, as a Kaiso-specific binding partner. GST pull-down assays confirmed that the interaction is mediated via the POZ domain of each protein, and co-immunoprecipitation experiments further supported an in vivo Kaiso-Znf131 interaction. Using a Cyclic Amplification and Selection of Targets (CAST) approach, we identified the 12-base pair DNA palindrome sequence GTCGCR-(X)(n)-YGCGAC as a potential Znf131 binding element (ZBE). In vitro studies using electrophoretic mobility shift assay (EMSA) demonstrated that Znf131 binds the ZBE via its zinc finger domain. Znf131 DNA-binding specificity was confirmed using competition assays and ZBE mutational analyses. An artificial promoter-reporter construct containing four tandem copies of the ZBE was constructed and used to assess Znf131 transcriptional properties. We observed dose-dependent transcriptional activation of this artificial promoter-reporter by Znf131 in both epithelial and fibroblast cells, suggesting that Znf131 is a transcriptional activator. Kaiso overexpression significantly decreased the Znf131-mediated transcriptional activation, and interestingly, co-expression of the Kaiso-specific interaction partner p120(ctn) relieved Kaiso's inhibition of Znf131-mediated transcriptional activation. These findings indicate that Znf131 is a transcriptional activator, a less common function of POZ-ZF proteins, that is negatively regulated by its heterodimerization partner Kaiso.

Sum frequency generation-compressive sensing microscope.

A new sum frequency generation imaging microscope using a novel sampling theory, compressive sensing (CS), has been developed for surface studies. CS differentiates itself from the conventional sampling methods by collecting fewer measurements than the traditional methods to reconstruct a high quality image. Pseudorandom patterns were applied to a light modulator and reflected the sum frequency (SF) signal generated from the sample into a photomultiplier tube detector. The image of the sample was reconstructed using sparsity preserving algorithms from the SF signal. The influences of the number of CS testing patterns applied and the number of SF pulses acquired for each pattern on the quality of the images was investigated and a comparison of the image quality with the traditional raster scan was made at varying resolutions for a gold patterned Si surface. Our results demonstrate the CS technique achieved 16 times the pixel density beyond the resolution where the raster scan strategy lost its ability to image the sample due to the dilution of the SF signal below the detection limit of the detector.

Hypoxia inducible factor (HIF)-2 alpha is required for the development of the catecholaminergic phenotype of sympathoadrenal cells.

The basic helix-loop-helix transcription factor, hypoxia inducible factor (HIF)-2alpha has been implicated in the development of the catecholaminergic phenotype in cells of the sympathoadrenal (SA) lineage; however, the underlying mechanisms and HIF-2alpha targets remain unclear. Using an immortalized rat adrenomedullary chromaffin cell line (MAH cells) derived from a fetal SA progenitor, we examined the role of HIF-2alpha in catecholamine biosynthesis. Chronic hypoxia (2% O(2), 24 h) induced HIF-2alpha in MAH cells but expression of the rate-limiting enzyme, tyrosine hydroxylase (TH) and catecholamine levels were unaltered. Interestingly, HIF-2alpha depleted MAH cells showed dramatically lower (5-12 times) levels of dopamine and noradrenaline compared with wild-type and scrambled controls, even in normoxia (21% O(2)). This was correlated with a marked reduction in the expression of DOPA decarboxylase (DDC) and dopamine beta hydroxylase (DbetaH) but not TH. Chromatin immunoprecipitation assays revealed that HIF-2alpha was bound to the DDC gene promoter which contains two putative hypoxia response elements. These data suggest that a basal level of HIF-2alpha function is required for the normal developmental expression of DDC and DbetaH in SA progenitor cells, and that loss of this function leads to impaired catecholamine biosynthesis.

Nuclear trafficking of the POZ-ZF protein Znf131.

Znf131 is a member of the BTB/POZ family of transcription factors with roles in development and carcinogenesis. Like many members of this protein family, Znf131 displays robust nuclear localization in cultured cells, but the mechanism(s) of Znf131 nuclear trafficking is unknown. Here, we report the mechanism of Znf131 nuclear localization. Visual inspection of the Znf131 amino acid sequence revealed three basic regions (BR-1, -2 and -3) with the potential to serve as nuclear localization signals (NLS). Of the three basic regions, only BR-1 functioned independently to efficiently target heterologous beta-gal-GFP fusion proteins to HeLa cell nuclei. However, a Znf131 truncation mutant containing BR-2 and BR-3 efficiently targeted heterologous beta-gal-GFP fusion proteins to HeLa cell nuclei. Mutational analysis of full-length GFP-tagged Znf131 revealed that loss of any one BR alone did not prevent Znf131 nuclear localization. This apparent redundancy in NLS activity was due to the fact that intact BR-1 or BR-2 alone could target full-length Znf131 to nuclei. Consequently, simultaneous mutation of BR-1 and BR-2 abolished full-length Znf131 nuclear localization. Therefore, BR-1 and BR-2 are functional NLSs for Znf131 and as such are designated NLS-1 and NLS-2. Finally, wild type Znf131, and not a Znf131 NLS-defective mutant (NLS-1m/NLS-2m) interacted preferentially with the nuclear import receptor Importin-alpha3 in vitro.

Graphite epoxide.

Oxidation of graphite may be carried out by reaction with meta-chloroperoxybenzoic acid to yield graphite epoxide. Scanning tunneling microscopy (STM) showed that the functionalization occurs at the edges rather than on the basal plane of the graphite. Quantification of the epoxide content is possible through the deepoxidation reaction using MeReO3/PPh3.

Synthesis of a nanocar with an angled chassis. Toward circling movement.

Nanocars with an angled chassis have been synthesized and imaged using scanning tunneling microscopy. These angled chassis nanocars were designed to further our understanding of the directional control and surface-rolling capabilities of this class of nanomachines. The alkylated carbazole inner core might enable the molecular scaffold to produce circular rolling motions of the nanovehicles on surfaces.

Synthesis of a porphyrin-fullerene pinwheel.

We disclose the synthesis of a porphyrin-fullerene pinwheel that was subsequently observed by scanning tunneling microscopy. The molecule was designed to further our understanding of fullerene-surface interactions, directional control, and surface-rolling versus pivoting capabilities of this class of nanomachines. The inner porphyrin provides the square planar configuration that might lead to realization of the pinwheel spiraling motion on surfaces.

Chemical Imaging of Self-Assembled Monolayers on Copper Using Compressive Hyperspectral Sum Frequency Generation Microscopy.

Sum frequency generation microscopy is a label-free optical imaging technique with intrinsic molecular vibrational contrast for surface studies. Recent developments of compressive sensing broad-band hyperspectral SFG microscopy have demonstrated the potential application for imaging monolayer at metal surfaces with micrometer spatial resolution. Here is presented the capability of chemical imaging of spatially patterned monolayers of 1-octadecanethiol (ODT) and 16-methoxy-1-hexadecanethiol (MeOHT) molecules assembled on a copper surface. The spatial distributions of the monolayer with vibrational-spectral contrast are well-demonstrated at different frequency regions through reconstruction of the hypercube using a 3-dimensional total variation regularization algorithm (3DTV). Spatial-chemical distributions of each component are also reconstructed directly from the compressive measurements by endmember unmixing (CEU) scheme. Compared to 3DTV algorithm, the reconstruction from CEU shows spatial distribution of each component on the surfaces, and demonstrates the ability to characterize the domains of mixed-molecules on surfaces.

Regulation of the rapsyn promoter by kaiso and delta-catenin.

Rapsyn is a synapse-specific protein that is required for clustering acetylcholine receptors at the neuromuscular junction. Analysis of the rapsyn promoter revealed a consensus site for the transcription factor Kaiso within a region that is mutated in a subset of patients with congenital myasthenic syndrome. Kaiso is a POZ-zinc finger family transcription factor which recognizes the specific core consensus sequence CTGCNA (where N is any nucleotide). Previously, the only known binding partner for Kaiso was the cell adhesion cofactor, p120 catenin. Here we show that delta-catenin, a brain-specific member of the p120 catenin subfamily, forms a complex with Kaiso. Antibodies against Kaiso and delta-catenin recognize proteins in the nuclei of C2C12 myocytes and at the postsynaptic domain of the mouse neuromuscular junction. Endogenous Kaiso in C2C12 cells coprecipitates with the rapsyn promoter in vivo as shown by chromatin immunoprecipitation assay. Minimal promoter assays demonstrated that the rapsyn promoter can be activated by Kaiso and delta-catenin; this activation is apparently muscle specific. These results provide the first experimental evidence that rapsyn is a direct sequence-specific target of Kaiso and delta-catenin. We propose a new model of synapse-specific transcription that involves the interaction of Kaiso, delta-catenin, and myogenic transcription factors at the neuromuscular junction.

Compressive Broad-Band Hyperspectral Sum Frequency Generation Microscopy to Study Functionalized Surfaces.

A broad-band sum frequency generation microscope has been developed for the study of molecular monolayers on surfaces. Because sum frequency generation is a vibrational spectroscopy based on a second-order optical process, it is uniquely sensitive to detecting a molecule's vibrational fingerprints specifically at interfaces. In this microscope, a structured illumination beam generated by a spatial light modulator is used to irradiate the sample with a series of sparsifying pseudorandom patterns. The spectra associated with each pattern are then input into a reconstruction algorithm to compressively recover the full hyperspectral image cube. As a proof-of-principle, this system performed molecule-specific imaging of a microcontact-printed self-assembled monolayer of alkanethiolate on copper. This hyperspectral compressive imaging effectively recovered both spatial and spectral surface features with compression greater than 80%, meaning more than a 5-fold decrease in acquisition time compared to traditional methods.

Super Temporal-Resolved Microscopy (STReM).

Super-resolution microscopy typically achieves high spatial resolution, but the temporal resolution remains low. We report super temporal-resolved microscopy (STReM) to improve the temporal resolution of 2D super-resolution microscopy by a factor of 20 compared to that of the traditional camera-limited frame rate. This is achieved by rotating a phase mask in the Fourier plane during data acquisition and then recovering the temporal information by fitting the point spread function (PSF) orientations. The feasibility of this technique is verified with both simulated and experimental 2D adsorption/desorption and 2D emitter transport. When STReM is applied to measure protein adsorption at a glass surface, previously unseen dynamics are revealed.

Generalized recovery algorithm for 3D super-resolution microscopy using rotating point spread functions.

Super-resolution microscopy with phase masks is a promising technique for 3D imaging and tracking. Due to the complexity of the resultant point spread functions, generalized recovery algorithms are still missing. We introduce a 3D super-resolution recovery algorithm that works for a variety of phase masks generating 3D point spread functions. A fast deconvolution process generates initial guesses, which are further refined by least squares fitting. Overfitting is suppressed using a machine learning determined threshold. Preliminary results on experimental data show that our algorithm can be used to super-localize 3D adsorption events within a porous polymer film and is useful for evaluating potential phase masks. Finally, we demonstrate that parallel computation on graphics processing units can reduce the processing time required for 3D recovery. Simulations reveal that, through desktop parallelization, the ultimate limit of real-time processing is possible. Our program is the first open source recovery program for generalized 3D recovery using rotating point spread functions.

AFM and STM characterization of thiol and thiophene functionalized SWNTs: pitfalls in the use of chemical markers to determine the extent of sidewall functionalization in SWNTs.

Thiol- and thiophene-functionalized SWNTs prepared via the reaction of a substituted amine with fluoronanotubes show similar levels of sidewall functionalization, however, the use of Au nanoparticles as chemical markers for AFM gives misleading results for substituent distribution since STM shows the thiol substituents grouped in bands while the thiophene substituents uniformly distributed along the SWNTs.