Integrated photonic encoder for low power and high-speed image processing.

Modern lens designs are capable of resolving greater than 10 gigapixels, while advances in camera frame-rate and hyperspectral imaging have made data acquisition rates of Terapixel/second a real possibility. The main bottlenecks preventing such high data-rate systems are power consumption and data storage. In this work, we show that analog photonic encoders could address this challenge, enabling high-speed image compression using orders-of-magnitude lower power than digital electronics. Our approach relies on a silicon-photonics front-end to compress raw image data, foregoing energy-intensive image conditioning and reducing data storage requirements. The compression scheme uses a passive disordered photonic structure to perform kernel-type random projections of the raw image data with minimal power consumption and low latency. A back-end neural network can then reconstruct the original images with structural similarity exceeding 90%. This scheme has the potential to process data streams exceeding Terapixel/second using less than 100 fJ/pixel, providing a path to ultra-high-resolution data and image acquisition systems.

Compressive video via IR-pulsed illumination.

We propose and demonstrate a compressive temporal imaging system based on pulsed illumination to encode temporal dynamics into the signal received by the imaging sensor during exposure time. Our approach enables >10x increase in effective frame rate without increasing camera complexity. To mitigate the complexity of the inverse problem during reconstruction, we introduce two keyframes: one before and one after the coded frame. We also craft what we believe to be a novel deep learning architecture for improved reconstruction of the high-speed scenes, combining specialized convolutional and transformer architectures. Simulation and experimental results clearly demonstrate the reconstruction of high-quality, high-speed videos from the compressed data.