Event-based dual photography for transparent scene reconstruction.

Light transport contains all light information between a light source and an image sensor. As an important application of light transport, dual photography has been a popular research topic, but it is challenged by long acquisition time, low signal-to-noise ratio, and the storage or processing of a large number of measurements. In this Letter, we propose a novel hardware setup that combines a flying-spot micro-electro mechanical system (MEMS) modulated projector with an event camera to implement dual photography for 3D scanning in both line-of-sight (LoS) and non-line-of-sight (NLoS) scenes with a transparent object. In particular, we achieved depth extraction from the LoS scenes and 3D reconstruction of the object in a NLoS scene using event light transport.

Deep camera obscura: an image restoration pipeline for pinhole photography.

Modern machine learning has enhanced the image quality for consumer and mobile photography through low-light denoising, high dynamic range (HDR) imaging, and improved demosaicing among other applications. While most of these advances have been made for normal lens-based cameras, there has been an emerging body of research for improved photography for lensless cameras using thin optics such as amplitude or phase masks, diffraction gratings, or diffusion layers. These lensless cameras are suited for size and cost-constrained applications such as tiny robotics and microscopy that prohibit the use of a large lens. However, the earliest and simplest camera design, the camera obscura or pinhole camera, has been relatively overlooked for machine learning pipelines with minimal research on enhancing pinhole camera images for everyday photography applications. In this paper, we develop an image restoration pipeline of the pinhole system to enhance the pinhole image quality through joint denoising and deblurring. Our pipeline integrates optics-based filtering and reblur losses for reconstructing high resolution still images (2600 × 1952) as well as temporal consistency for video reconstruction to enable practical exposure times (30 FPS) for high resolution video (1920 × 1080). We demonstrate high 2D image quality on real pinhole images that is on-par or slightly improved compared to other lensless cameras. This work opens up the potential of pinhole cameras to be used for photography in size-limited devices such as smartphones in the future.

Turbulence strength C n2 estimation from video using physics-based deep learning.

Images captured from a long distance suffer from dynamic image distortion due to turbulent flow of air cells with random temperatures, and thus refractive indices. This phenomenon, known as image dancing, is commonly characterized by its refractive-index structure constant C n2 as a measure of the turbulence strength. For many applications such as atmospheric forecast model, long-range/astronomy imaging, and aviation safety, optical communication technology, C n2 estimation is critical for accurately sensing the turbulent environment. Previous methods for C n2 estimation include estimation from meteorological data (temperature, relative humidity, wind shear, etc.) for single-point measurements, two-ended pathlength measurements from optical scintillometer for path-averaged C n2, and more recently estimating C n2 from passive video cameras for low cost and hardware complexity. In this paper, we present a comparative analysis of classical image gradient methods for C n2 estimation and modern deep learning-based methods leveraging convolutional neural networks. To enable this, we collect a dataset of video capture along with reference scintillometer measurements for ground truth, and we release this unique dataset to the scientific community. We observe that deep learning methods can achieve higher accuracy when trained on similar data, but suffer from generalization errors to other, unseen imagery as compared to classical methods. To overcome this trade-off, we present a novel physics-based network architecture that combines learned convolutional layers with a differentiable image gradient method that maintains high accuracy while being generalizable across image datasets.

Dense Lissajous sampling and interpolation for dynamic light-transport.

Light-transport represents the complex interactions of light in a scene. Fast, compressed, and accurate light-transport capture for dynamic scenes is an open challenge in vision and graphics. In this paper, we integrate the classical idea of Lissajous sampling with novel control strategies for dynamic light-transport applications such as relighting water drops and seeing around corners. In particular, this paper introduces an improved Lissajous projector hardware design and discusses calibration and capture for a microelectromechanical (MEMS) mirror-based projector. Further, we show progress towards speeding up the hardware-based Lissajous subsampling for dual light transport frames, and investigate interpolation algorithms for recovering back the missing data. Our captured dynamic light transport results show complex light scattering effects for dense angular sampling, and we also show dual non-line-of-sight (NLoS) capture of dynamic scenes. This work is the first step towards adaptive Lissajous control for dynamic light-transport.

Altered speech patterns in subjects with post-traumatic headache due to mild traumatic brain injury.

BACKGROUND/OBJECTIVE: Changes in speech can be detected objectively before and during migraine attacks. The goal of this study was to interrogate whether speech changes can be detected in subjects with post-traumatic headache (PTH) attributed to mild traumatic brain injury (mTBI) and whether there are within-subject changes in speech during headaches compared to the headache-free state. METHODS: Using a series of speech elicitation tasks uploaded via a mobile application, PTH subjects and healthy controls (HC) provided speech samples once every 3 days, over a period of 12 weeks. The following speech parameters were assessed: vowel space area, vowel articulation precision, consonant articulation precision, average pitch, pitch variance, speaking rate and pause rate. Speech samples of subjects with PTH were compared to HC. To assess speech changes associated with PTH, speech samples of subjects during headache were compared to speech samples when subjects were headache-free. All analyses were conducted using a mixed-effect model design. RESULTS: Longitudinal speech samples were collected from nineteen subjects with PTH (mean age = 42.5, SD = 13.7) who were an average of 14 days (SD = 32.2) from their mTBI at the time of enrollment and thirty-one HC (mean age = 38.7, SD = 12.5). Regardless of headache presence or absence, PTH subjects had longer pause rates and reductions in vowel and consonant articulation precision relative to HC. On days when speech was collected during a headache, there were longer pause rates, slower sentence speaking rates and less precise consonant articulation compared to the speech production of HC. During headache, PTH subjects had slower speaking rates yet more precise vowel articulation compared to when they were headache-free. CONCLUSIONS: Compared to HC, subjects with acute PTH demonstrate altered speech as measured by objective features of speech production. For individuals with PTH, speech production may have been more effortful resulting in slower speaking rates and more precise vowel articulation during headache vs. when they were headache-free, suggesting that speech alterations were related to PTH and not solely due to the underlying mTBI.

Dual light field and polarization imaging using CMOS diffractive image sensors.

In this Letter we present, to the best of our knowledge, the first integrated CMOS image sensor that can simultaneously perform light field and polarization imaging without the use of external filters or additional optical elements. Previous work has shown how photodetectors with two stacks of integrated metal gratings above them (called angle sensitive pixels) diffract light in a Talbot pattern to capture four-dimensional light fields. We show, in addition to diffractive imaging, that these gratings polarize incoming light and characterize the response of these sensors to polarization and incidence angle. Finally, we show two applications of polarization imaging: imaging stress-induced birefringence and identifying specular reflections in scenes to improve light field algorithms for these scenes.

Robust Vocal Quality Feature Embeddings for Dysphonic Voice Detection.

Approximately 1.2% of the world's population has impaired voice production. As a result, automatic dysphonic voice detection has attracted considerable academic and clinical interest. However, existing methods for automated voice assessment often fail to generalize outside the training conditions or to other related applications. In this paper, we propose a deep learning framework for generating acoustic feature embeddings sensitive to vocal quality and robust across different corpora. A contrastive loss is combined with a classification loss to train our deep learning model jointly. Data warping methods are used on input voice samples to improve the robustness of our method. Empirical results demonstrate that our method not only achieves high in-corpus and cross-corpus classification accuracy but also generates good embeddings sensitive to voice quality and robust across different corpora. We also compare our results against three baseline methods on clean and three variations of deteriorated in-corpus and cross-corpus datasets and demonstrate that the proposed model consistently outperforms the baseline methods.

Effects of time-dependent stimuli in a competitive neural network model of perceptual rivalry.

We analyze a competitive neural network model of perceptual rivalry that receives time-varying inputs. Time-dependence of inputs can be discrete or smooth. Spike frequency adaptation provides negative feedback that generates network oscillations when inputs are constant in time. Oscillations that resemble perceptual rivalry involve only one population being "ON" at a time, which represents the dominance of a single percept at a time. As shown in Laing and Chow (J. Comput. Neurosci. 12(1):39­53, 2002), for sufficiently high contrast, one can derive relationships between dominance times and contrast that agree with Levelt's propositions (Levelt in On binocular rivalry, 1965). Time-dependent stimuli give rise to novel network oscillations where both, one, or neither populations are "ON" at any given time. When a single population receives an interrupted stimulus, the fundamental mode of behavior we find is phase-locking, where the temporally driven population locks its state to the stimulus. Other behaviors are analyzed as bifurcations from this forced oscillation, using fast/slow analysis that exploits the slow timescale of adaptation. When both populations receive time-varying input, we find mixtures of fusion and sole population dominance, and we partition parameter space into particular oscillation types. Finally, when a single population's input contrast is smoothly varied in time, 1:n mode-locked states arise through period-adding bifurcations beyond phase-locking. Our results provide several testable predictions for future psychophysical experiments on perceptual rivalry.

Programmable Non-Epipolar Indirect Light Transport: Capture and Analysis.

The decomposition of light transport into direct and global components, diffuse and specular interreflections, and subsurface scattering allows for new visualizations of light in everyday scenes. In particular, indirect light contains a myriad of information about the complex appearance of materials useful for computer vision and inverse rendering applications. In this paper, we present a new imaging technique that captures and analyzes components of indirect light via light transport using a synchronized projector-camera system. The rectified system illuminates the scene with epipolar planes corresponding to projector rows, and we vary two key parameters to capture plane-to-ray light transport between projector row and camera pixel: (1) the offset between projector row and camera row in the rolling shutter (implemented as synchronization delay), and (2) the exposure of the camera row. We describe how this synchronized rolling shutter performs illumination multiplexing, and develop a nonlinear optimization algorithm to demultiplex the resulting 3D light transport operator. Using our system, we are able to capture live short and long-range non-epipolar indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties. In particular, we show the utility of indirect imaging for capturing and analyzing the hidden structure of veins in human skin.

Changing Cycle Lengths in State-Transition Models: Challenges and Solutions.

The choice of a cycle length in state-transition models should be determined by the frequency of clinical events and interventions. Sometimes there is need to decrease the cycle length of an existing state-transition model to reduce error in outcomes resulting from discretization of the underlying continuous-time phenomena or to increase the cycle length to gain computational efficiency. Cycle length conversion is also frequently required if a new state-transition model is built using observational data that have a different measurement interval than the model's cycle length. We show that a commonly used method of converting transition probabilities to different cycle lengths is incorrect and can provide imprecise estimates of model outcomes. We present an accurate approach that is based on finding the root of a transition probability matrix using eigendecomposition. We present underlying mathematical challenges of converting cycle length in state-transition models and provide numerical approximation methods when the eigendecomposition method fails. Several examples and analytical proofs show that our approach is more general and leads to more accurate estimates of model outcomes than the commonly used approach. MATLAB codes and a user-friendly online toolkit are made available for the implementation of the proposed methods.