Deep Learning-Based Target Tracking and Classification for Low Quality Videos Using Coded Aperture Cameras.

Compressive sensing has seen many applications in recent years. One type of compressive sensing device is the Pixel-wise Code Exposure (PCE) camera, which has low power consumption and individual control of pixel exposure time. In order to use PCE cameras for practical applications, a time consuming and lossy process is needed to reconstruct the original frames. In this paper, we present a deep learning approach that directly performs target tracking and classification in the compressive measurement domain without any frame reconstruction. In particular, we propose to apply You Only Look Once (YOLO) to detect and track targets in the frames and we propose to apply Residual Network (ResNet) for classification. Extensive simulations using low quality optical and mid-wave infrared (MWIR) videos in the SENSIAC database demonstrated the efficacy of our proposed approach.

Compact all-CMOS spatiotemporal compressive sensing video camera with pixel-wise coded exposure.

We present a low power all-CMOS implementation of temporal compressive sensing with pixel-wise coded exposure. This image sensor can increase video pixel resolution and frame rate simultaneously while reducing data readout speed. Compared to previous architectures, this system modulates pixel exposure at the individual photo-diode electronically without external optical components. Thus, the system provides reduction in size and power compare to previous optics based implementations. The prototype image sensor (127 × 90 pixels) can reconstruct 100 fps videos from coded images sampled at 5 fps. With 20× reduction in readout speed, our CMOS image sensor only consumes 14µW to provide 100 fps videos.

Pix2HDR - A pixel-wise acquisition and deep learning-based synthesis approach for high-speed HDR videos.

Accurately capturing dynamic scenes with wideranging motion and light intensity is crucial for many vision applications. However, acquiring high-speed high dynamic range (HDR) video is challenging because the camera's frame rate restricts its dynamic range. Existing methods sacrifice speed to acquire multi-exposure frames. Yet, misaligned motion in these frames can still pose complications for HDR fusion algorithms, resulting in artifacts. Instead of frame-based exposures, we sample the videos using individual pixels at varying exposures and phase offsets. Implemented on a monochrome pixel-wise programmable image sensor, our sampling pattern captures fast motion at a high dynamic range. We then transform pixel-wise outputs into an HDR video using end-to-end learned weights from deep neural networks, achieving high spatiotemporal resolution with minimized motion blurring. We demonstrate aliasing-free HDR video acquisition at 1000 FPS, resolving fast motion under low-light conditions and against bright backgrounds - both challenging conditions for conventional cameras. By combining the versatility of pixel-wise sampling patterns with the strength of deep neural networks at decoding complex scenes, our method greatly enhances the vision system's adaptability and performance in dynamic conditions.

Pixel-wise programmability enables dynamic high-SNR cameras for high-speed microscopy.

High-speed wide-field fluorescence microscopy has the potential to capture biological processes with exceptional spatiotemporal resolution. However, conventional cameras suffer from low signal-to-noise ratio at high frame rates, limiting their ability to detect faint fluorescent events. Here, we introduce an image sensor where each pixel has individually programmable sampling speed and phase, so that pixels can be arranged to simultaneously sample at high speed with a high signal-to-noise ratio. In high-speed voltage imaging experiments, our image sensor significantly increases the output signal-to-noise ratio compared to a low-noise scientific CMOS camera (~2-3 folds). This signal-to-noise ratio gain enables the detection of weak neuronal action potentials and subthreshold activities missed by the standard scientific CMOS cameras. Our camera with flexible pixel exposure configurations offers versatile sampling strategies to improve signal quality in various experimental conditions.

The influence of stereopsis on visual saliency in a proto-object based model of selective attention.

Some animals including humans use stereoscopic vision which reconstructs spatial information about the environment from the disparity between images captured by eyes in two separate adjacent locations. Like other sensory information, such stereoscopic information is expected to influence attentional selection. We develop a biologically plausible model of binocular vision to study its effect on bottom-up visual attention, i.e., visual saliency. In our model, the scene is organized in terms of proto-objects on which attention acts, rather than on unbound sets of elementary features. We show that taking into account the stereoscopic information improves the performance of the model in the prediction of human eye movements with statistically significant differences.

Neuromorphic applications in medicine.

In recent years, there has been a growing demand for miniaturization, low power consumption, quick treatments, and non-invasive clinical strategies in the healthcare industry. To meet these demands, healthcare professionals are seeking new technological paradigms that can improve diagnostic accuracy while ensuring patient compliance. Neuromorphic engineering, which uses neural models in hardware and software to replicate brain-like behaviors, can help usher in a new era of medicine by delivering low power, low latency, small footprint, and high bandwidth solutions. This paper provides an overview of recent neuromorphic advancements in medicine, including medical imaging and cancer diagnosis, processing of biosignals for diagnosis, and biomedical interfaces, such as motor, cognitive, and perception prostheses. For each section, we provide examples of how brain-inspired models can successfully compete with conventional artificial intelligence algorithms, demonstrating the potential of neuromorphic engineering to meet demands and improve patient outcomes. Lastly, we discuss current struggles in fitting neuromorphic hardware with non-neuromorphic technologies and propose potential solutions for future bottlenecks in hardware compatibility.

Pixel-wise programmability enables dynamic high-SNR cameras for high-speed microscopy.

High-speed widefield fluorescence microscopy has the potential to capture biological processes with exceptional spatiotemporal resolution. However, conventional cameras suffer from low signal-to-noise ratio (SNR) at high frame rates, limiting their ability to detect faint fluorescent events. Here we introduce an image sensor where each pixel has individually programmable sampling speed and phase, so that pixels can be arranged to simultaneously sample at high speed with a high SNR. In high-speed voltage imaging experiments, our image sensor significantly increases the output SNR compared to a low-noise scientific CMOS camera (~2-3 folds). This SNR gain enables the detection of weak neuronal action potentials and subthreshold activities missed by the standard scientific CMOS cameras. Our proposed camera with flexible pixel exposure configurations offers versatile sampling strategies to improve signal quality in various experimental conditions.

First Organoid Intelligence (OI) workshop to form an OI community.

The brain is arguably the most powerful computation system known. It is extremely efficient in processing large amounts of information and can discern signals from noise, adapt, and filter faulty information all while running on only 20 watts of power. The human brain's processing efficiency, progressive learning, and plasticity are unmatched by any computer system. Recent advances in stem cell technology have elevated the field of cell culture to higher levels of complexity, such as the development of three-dimensional (3D) brain organoids that recapitulate human brain functionality better than traditional monolayer cell systems. Organoid Intelligence (OI) aims to harness the innate biological capabilities of brain organoids for biocomputing and synthetic intelligence by interfacing them with computer technology. With the latest strides in stem cell technology, bioengineering, and machine learning, we can explore the ability of brain organoids to compute, and store given information (input), execute a task (output), and study how this affects the structural and functional connections in the organoids themselves. Furthermore, understanding how learning generates and changes patterns of connectivity in organoids can shed light on the early stages of cognition in the human brain. Investigating and understanding these concepts is an enormous, multidisciplinary endeavor that necessitates the engagement of both the scientific community and the public. Thus, on Feb 22-24 of 2022, the Johns Hopkins University held the first Organoid Intelligence Workshop to form an OI Community and to lay out the groundwork for the establishment of OI as a new scientific discipline. The potential of OI to revolutionize computing, neurological research, and drug development was discussed, along with a vision and roadmap for its development over the coming decade.

Modified map-seeking circuit: use of computer-aided detection in locating postoperative retained foreign bodies.

BACKGROUND: More than 98% of intra-operative X-rays taken to search for postoperative retained foreign bodies (RFBs) have negative findings; in over 30% of cases of such X-rays, the finding is a false negative. Newer technologies created to find RFBs must not only reduce the false-negative rate, but also must not increase the burden of detecting RFBs. We have introduced the use of computer-aided detection (CAD) to facilitate the detection of RFBs on X-rays utilizing a modified version of map-seeking circuit (MSC) algorithm the referenced map-seeking circuit (RMSC), for our proof-of-concept study for detection of needles in plain abdominal X-rays. METHODS: Images were obtained by using a portable cassette-based X-ray machine and a C-arm (digital) machine, both of which are commonly used in the operating room. The images obtained using these machines were divided into subimages of approximately 250 × 250 pixels each, for a total of 455 subimages from the cassette-based machine (A) and 365 from the digital machine (B) for use as test samples. Images obtained from A and B were analyzed separately using our modified MSC algorithm with a minimum (τ = 0) and a maximum threshold (τ = 0.5). RESULTS: The automated detection rate (positive predictive value) was 86%, with a false positive/negative rate of 10% to 15% when τ was zero. CONCLUSION: The CAD-based RMSC algorithm has the potential to improve the accuracy with which RFBs can be found in X-rays. Further research is needed to optimize the detection rate and to identify a wider range of RFBs.

Analysis and Design Methodology of RF Energy Harvesting Rectifier Circuit for Ultra-Low Power Applications.

This paper reviews and analyses the design of popular radio frequency energy harvesting systems and proposes a method to qualitatively and quantitatively analyze their circuit architectures using new square-wave approximation method. This approach helps in simplifying design analysis. Using this analysis, we can establish no load output voltage characteristics, upper limit on rectifier efficiency, and maximum power characteristics of a rectifier. This paper will help guide the design of RF energy harvesting rectifier circuits for radio frequency identification (RFIDs), the Internet of Things (IoTs), wearable, and implantable medical device applications. Different application scenarios are explained in the context of design challenges, and corresponding design considerations are discussed in order to evaluate their performance. The pros and cons of different rectifier topologies are also investigated. In addition to presenting the popular rectifier topologies, new measurement results of these energy harvester topologies, fabricated in 65nm, 130nm and 180nm CMOS technologies are also presented.

Estimating parameters of generalized integrate-and-fire neurons from the maximum likelihood of spike trains.

When a neuronal spike train is observed, what can we deduce from it about the properties of the neuron that generated it? A natural way to answer this question is to make an assumption about the type of neuron, select an appropriate model for this type, and then choose the model parameters as those that are most likely to generate the observed spike train. This is the maximum likelihood method. If the neuron obeys simple integrate-and-fire dynamics, Paninski, Pillow, and Simoncelli (2004) showed that its negative log-likelihood function is convex and that, at least in principle, its unique global minimum can thus be found by gradient descent techniques. Many biological neurons are, however, known to generate a richer repertoire of spiking behaviors than can be explained in a simple integrate-and-fire model. For instance, such a model retains only an implicit (through spike-induced currents), not an explicit, memory of its input; an example of a physiological situation that cannot be explained is the absence of firing if the input current is increased very slowly. Therefore, we use an expanded model (Mihalas & Niebur, 2009 ), which is capable of generating a large number of complex firing patterns while still being linear. Linearity is important because it maintains the distribution of the random variables and still allows maximum likelihood methods to be used. In this study, we show that although convexity of the negative log-likelihood function is not guaranteed for this model, the minimum of this function yields a good estimate for the model parameters, in particular if the noise level is treated as a free parameter. Furthermore, we show that a nonlinear function minimization method (r-algorithm with space dilation) usually reaches the global minimum.

Sensor and display human factors based design constraints for head mounted and tele-operation systems.

For mobile imaging systems in head mounted displays and tele-operation systems it is important to maximize the amount of visual information transmitted to the human visual system without exceeding its input capacity. This paper aims to describe the design constraints on the imager and display systems of head mounted devices and tele-operated systems based upon the capabilities of the human visual system. We also present the experimental results of methods to improve the amount of visual information conveyed to a user when trying to display a high dynamic range image on a low dynamic range display.

A Neuromorphic Proto-Object Based Dynamic Visual Saliency Model With a Hybrid FPGA Implementation.

Computing and attending to salient regions of a visual scene is an innate and necessary preprocessing step for both biological and engineered systems performing high-level visual tasks including object detection, tracking, and classification. Computational bandwidth and speed are improved by preferentially devoting computational resources to salient regions of the visual field. The human brain computes saliency effortlessly, but modeling this task in engineered systems is challenging. We first present a neuromorphic dynamic saliency model, which is bottom-up, feed-forward, and based on the notion of proto-objects with neurophysiological spatio-temporal features requiring no training. Our neuromorphic model outperforms state-of-the-art dynamic visual saliency models in predicting human eye fixations (i.e., ground truth saliency). Secondly, we present a hybrid FPGA implementation of the model for real-time applications, capable of processing 112×84 resolution frames at 18.71 Hz running at a 100 MHz clock rate - a 23.77× speedup from the software implementation. Additionally, our fixed-point model of the FPGA implementation yields comparable results to the software implementation.

Decoding accelerometry for classification and prediction of critically ill patients with severe brain injury.

Our goal is to explore quantitative motor features in critically ill patients with severe brain injury (SBI). We hypothesized that computational decoding of these features would yield information on underlying neurological states and outcomes. Using wearable microsensors placed on all extremities, we recorded a median 24.1 (IQR: 22.8-25.1) hours of high-frequency accelerometry data per patient from a prospective cohort (n = 69) admitted to the ICU with SBI. Models were trained using time-, frequency-, and wavelet-domain features and levels of responsiveness and outcome as labels. The two primary tasks were detection of levels of responsiveness, assessed by motor sub-score of the Glasgow Coma Scale (GCSm), and prediction of functional outcome at discharge, measured with the Glasgow Outcome Scale-Extended (GOSE). Detection models achieved significant (AUC: 0.70 [95% CI: 0.53-0.85]) and consistent (observation windows: 12 min-9 h) discrimination of SBI patients capable of purposeful movement (GCSm > 4). Prediction models accurately discriminated patients of upper moderate disability or better (GOSE > 5) with 2-6 h of observation (AUC: 0.82 [95% CI: 0.75-0.90]). Results suggest that time series analysis of motor activity yields clinically relevant insights on underlying functional states and short-term outcomes in patients with SBI.

A syndromic surveillance tool to detect anomalous clusters of COVID-19 symptoms in the United States.

Coronavirus SARS-COV-2 infections continue to spread across the world, yet effective large-scale disease detection and prediction remain limited. COVID Control: A Johns Hopkins University Study, is a novel syndromic surveillance approach, which collects body temperature and COVID-like illness (CLI) symptoms across the US using a smartphone app and applies spatio-temporal clustering techniques and cross-correlation analysis to create maps of abnormal symptomatology incidence that are made publicly available. The results of the cross-correlation analysis identify optimal temporal lags between symptoms and a range of COVID-19 outcomes, with new taste/smell loss showing the highest correlations. We also identified temporal clusters of change in taste/smell entries and confirmed COVID-19 incidence in Baltimore City and County. Further, we utilized an extended simulated dataset to showcase our analytics in Maryland. The resulting clusters can serve as indicators of emerging COVID-19 outbreaks, and support syndromic surveillance as an early warning system for disease prevention and control.

Proto-Object Based Saliency Model With Texture Detection Channel.

The amount of visual information projected from the retina to the brain exceeds the information processing capacity of the latter. Attention, therefore, functions as a filter to highlight important information at multiple stages of the visual pathway that requires further and more detailed analysis. Among other functions, this determines where to fixate since only the fovea allows for high resolution imaging. Visual saliency modeling, i.e. understanding how the brain selects important information to analyze further and to determine where to fixate next, is an important research topic in computational neuroscience and computer vision. Most existing bottom-up saliency models use low-level features such as intensity and color, while some models employ high-level features, like faces. However, little consideration has been given to mid-level features, such as texture, for visual saliency models. In this paper, we extend a biologically plausible proto-object based saliency model by adding simple texture channels which employ nonlinear operations that mimic the processing performed by primate visual cortex. The extended model shows statistically significant improved performance in predicting human fixations compared to the previous model. Comparing the performance of our model with others on publicly available benchmarking datasets, we find that our biologically plausible model matches the performance of other models, even though those were designed entirely for maximal performance with little regard to biological realism.

Closed-loop bioelectronic medicine for diabetes management.

Modulation of the nervous system by delivering electrical or pharmaceutical agents has contributed to the development of novel treatments to serious health disorders. Recent advances in multidisciplinary research has enabled the emergence of a new powerful therapeutic approach called bioelectronic medicine. Bioelectronic medicine exploits the fact that every organ in our bodies is neurally innervated and thus electrical interfacing with peripheral nerves can be a potential pathway for diagnosing or treating diseases such as diabetes. In this context, a plethora of studies have confirmed the important role of the nervous system in maintaining a tight regulation of glucose homeostasis. This has initiated new research exploring the opportunities of bioelectronic medicine for improving glucose control in people with diabetes, including regulation of gastric emptying, insulin sensitivity, and secretion of pancreatic hormones. Moreover, the development of novel closed-loop strategies aims to provide effective, specific and safe interfacing with the nervous system, and thereby targeting the organ of interest. This is especially valuable in the context of chronic diseases such as diabetes, where closed-loop bioelectronic medicine promises to provide real-time, autonomous and patient-specific therapies. In this article, we present an overview of the state-of-the-art for closed-loop neuromodulation systems in relation to diabetes and discuss future related opportunities for management of this chronic disease.

A Compact Free-Floating Device for Passive Charge-Balanced Neural Stimulation using PEDOT/CNT microelectrodes.

Wirelessly powered implants are increasingly being developed as free-floating single-channel devices to interface with neurons directly at stimulation sites. In order to stimulate neurons in a manner that is safe to both the electrode and the surrounding tissue, charge accumulation over time needs to be avoided. The implementation of conventional charge balancing methods often leads to an increase in system complexity, power consumption or area, all of which are critical parameters in ultra-small wireless devices. The proposed charge balancing method described in this work, which relies on bipolar capacitive integrated electrodes, does not increase these parameters. The standalone wirelessly powered stimulating implant is implemented in a 130nm CMOS technology and measures 0.009 mm3.

Detection and Confirmation of Multiple Human Targets Using Pixel-Wise Code Aperture Measurements.

Compressive video measurements can save bandwidth and data storage. However, conventional approaches to target detection require the compressive measurements to be reconstructed before any detectors are applied. This is not only time consuming but also may lose information in the reconstruction process. In this paper, we summarized the application of a recent approach to vehicle detection and classification directly in the compressive measurement domain to human targets. The raw videos were collected using a pixel-wise code exposure (PCE) camera, which condensed multiple frames into one frame. A combination of two deep learning-based algorithms (you only look once (YOLO) and residual network (ResNet)) was used for detection and confirmation. Optical and mid-wave infrared (MWIR) videos from a well-known database (SENSIAC) were used in our experiments. Extensive experiments demonstrated that the proposed framework was feasible for target detection up to 1500 m, but target confirmation needs more research.

A Miniature Wireless Silicon-on-Insulator Image Sensor for Brain Fluorescence Imaging.

Optical recording of genetically encoded calcium indicator (GECI) allows neuroscientists to study the activity of genetically labeled neuron populations, but our current tools lack the resolution, stability and are often too invasive. Here we present the design concepts, prototypes, and preliminary measurement results of a super-miniaturized wireless image sensor built using a 32nm Silicon-on-Insulator process. SOI process is optimal for wireless applications, and we can further thin the substrate to reduce overall device thickness to ~25µm and operate the pixels using back-side illumination. The proposed device is 300µm × 300µm. Our prototype is built on a 3 × 3mm die.