Bioinspired In-Sensor Multimodal Fusion for Enhanced Spatial and Spatiotemporal Association.

Multimodal perception can capture more precise and comprehensive information compared with unimodal approaches. However, current sensory systems typically merge multimodal signals at computing terminals following parallel processing and transmission, which results in the potential loss of spatial association information and requires time stamps to maintain temporal coherence for time-series data. Here we demonstrate bioinspired in-sensor multimodal fusion, which effectively enhances comprehensive perception and reduces the level of data transfer between sensory terminal and computation units. By adopting floating gate phototransistors with reconfigurable photoresponse plasticity, we realize the agile spatial and spatiotemporal fusion under nonvolatile and volatile photoresponse modes. To realize an optimal spatial estimation, we integrate spatial information from visual-tactile signals. For dynamic events, we capture and fuse in real time spatiotemporal information from visual-audio signals, realizing a dance-music synchronization recognition task without a time-stamping process. This in-sensor multimodal fusion approach provides the potential to simplify the multimodal integration system, extending the in-sensor computing paradigm.

Recent Progress in Wearable Near-Sensor and In-Sensor Intelligent Perception Systems.

As the Internet of Things (IoT) becomes more widespread, wearable smart systems will begin to be used in a variety of applications in people's daily lives, not only requiring the devices to have excellent flexibility and biocompatibility, but also taking into account redundant data and communication delays due to the use of a large number of sensors. Fortunately, the emerging paradigms of near-sensor and in-sensor computing, together with the proposal of flexible neuromorphic devices, provides a viable solution for the application of intelligent low-power wearable devices. Therefore, wearable smart systems based on new computing paradigms are of great research value. This review discusses the research status of a flexible five-sense sensing system based on near-sensor and in-sensor architectures, considering material design, structural design and circuit design. Furthermore, we summarize challenging problems that need to be solved and provide an outlook on the potential applications of intelligent wearable devices.

A Three-Dimensional Neuromorphic Photosensor Array for Nonvolatile In-Sensor Computing.

In-sensor computing hardware based on emerging reconfigurable photosensors can effectively reduce redundant data and decrease power consumption, which can greatly promote the evolution of machine vision. However, because of the complex device structures and low integration abilities, the common architectures mainly lie in two dimensions, resulting in low time and area efficiencies. Here we propose a three-dimensional (3D) neuromorphic photosensor array for parallel in-sensor image processing. It is constructed on a vertical Graphite/CuInP2S6/Graphite photosensor unit, where the directional Cu+ ion migrations after voltage pulse programming enable a reconfigurable photovoltaic effect and an in-sensor computing capability. With a memristor-like device structure, van der Waals interfaces, and a high uniformity with a low crosstalk problem, a 10 × 10 array is fabricated for intelligent image recognition. Furthermore, using a vertically stacked 3D 3 × 3 × 3 array, we demonstrate an in-sensor convolution strategy with high time and area efficiencies.

Biocompatible Material-Based Flexible Biosensors: From Materials Design to Wearable/Implantable Devices and Integrated Sensing Systems.

Human beings have a greater need to pursue life and manage personal or family health in the context of the rapid growth of artificial intelligence, big data, the Internet of Things, and 5G/6G technologies. The application of micro biosensing devices is crucial in connecting technology and personalized medicine. Here, the progress and current status from biocompatible inorganic materials to organic materials and composites are reviewed and the material-to-device processing is described. Next, the operating principles of pressure, chemical, optical, and temperature sensors are dissected and the application of these flexible biosensors in wearable/implantable devices is discussed. Different biosensing systems acting in vivo and in vitro, including signal communication and energy supply are then illustrated. The potential of in-sensor computing for applications in sensing systems is also discussed. Finally, some essential needs for commercial translation are highlighted and future opportunities for flexible biosensors are considered.

Integrated In-Sensor Computing Optoelectronic Device for Environment-Adaptable Artificial Retina Perception Application.

With the development and application of artificial intelligence, there is an appeal to the exploitation of various sensors and memories. As the most important perception of human beings, vision occupies more than 80% of all the received information. Inspired by biological eyes, an artificial retina based on 2D Janus MoSSe was fabricated, which could simulate functions of visual perception with electronic/ion and optical comodulation. Furthermore, inspired by human brain, sensing, memory, and neuromorphic computing functions were integrated on one device for multifunctional intelligent electronics, which was beneficial for scalability and high efficiency. Through the formation of faradic electric double layer (EDL) at the metal-oxide/electrolyte interfaces could realize synaptic weight changes. On the basis of the optoelectronic performances, light adaptation of biological eyes, preprocessing, and recognition of handwritten digits were implemented successfully. This work may provide a strategy for the future integrated sensing-memory-processing device for optoelectronic artificial retina perception application.

Three-Terminal Ovonic Threshold Switch (3T-OTS) with Tunable Threshold Voltage for Versatile Artificial Sensory Neurons.

Inspired by information processing in biological systems, sensor-combined edge-computing systems attract attention requesting artificial sensory neurons as essential ingredients. Here, we introduce a simple and versatile structure of artificial sensory neurons based on a novel three-terminal Ovonic threshold switch (3T-OTS), which features an electrically controllable threshold voltage (Vth). Combined with a sensor driving an output voltage, this 3T-OTS generates spikes with a frequency depending on an external stimulus. As a proof of concept, we have built an artificial retinal ganglion cell (RGC) by combining a 3T-OTS and a photodiode. Furthermore, this artificial RGC is combined with the reservoir-computing technique to perform a classification of chest X-ray images for normal, viral pneumonia, and COVID-19 infections, releasing the recognition accuracy of about 86.5%. These results indicate that the 3T-OTS is highly promising for applications in neuromorphic sensory systems, providing a building block for energy-efficient in-sensor computing devices.

Bio-Inspired In-Sensor Compression and Computing Based on Phototransistors.

The biological nervous system possesses a powerful information processing capability, and only needs a partial signal stimulation to perceive the entire signal. Likewise, the hardware implementation of an information processing system with similar capabilities is of great significance, for reducing the dimensions of data from sensors and improving the processing efficiency. Here, it is reported that indium-gallium-zinc-oxide thin film phototransistors exhibit the optoelectronic switching and light-tunable synaptic characteristics for in-sensor compression and computing. Phototransistor arrays can compress the signal while sensing, to realize in-sensor compression. Additionally, a reservoir computing network can also be implemented via phototransistors for in-sensor computing. By integrating these two systems, a neuromorphic system for high-efficiency in-sensor compression and computing is demonstrated. The results reveal that even for cases where the signal is compressed by 50%, the recognition accuracy of reconstructed signal still reaches ≈96%. The work paves the way for efficient information processing of human-computer interactions and the Internet of Things.

Development of Photonic In-Sensor Computing Based on a Mid-Infrared Silicon Waveguide Platform.

Neuromorphic in-sensor computing has provided an energy-efficient solution to smart sensor design and on-chip data processing. In recent years, various free-space-configured optoelectronic chips have been demonstrated for on-chip neuromorphic vision processing. However, on-chip waveguide-based in-sensor computing with different data modalities is still lacking. Here, by integrating a responsivity-tunable graphene photodetector onto the silicon waveguide, an on-chip waveguide-based in-sensor processing unit is realized in the mid-infrared wavelength range. The weighting operation is achieved by dynamically tuning the bias of the photodetector, which could reach 4 bit weighting precision. Three different neural network tasks are performed to demonstrate the capabilities of our device. First, image preprocessing is performed for handwritten digits and fashion product classification as a general task. Next, resistive-type glove sensor signals are reversed and applied to the photodetector as an input for gesture recognition. Finally, spectroscopic data processing for binary gas mixture classification is demonstrated by utilizing the broadband performance of the device from 3.65 to 3.8 µm. By extending the wavelength from near-infrared to mid-infrared, our work shows the capability of a waveguide-integrated tunable graphene photodetector as a viable weighting solution for photonic in-sensor computing. Furthermore, such a solution could be used for large-scale neuromorphic in-sensor computing in photonic integrated circuits at the edge.

Ultralow Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision.

In-sensor and near-sensor computing architectures enable multiply accumulate operations to be carried out directly at the point of sensing. In-sensor architectures offer dramatic power and speed improvements over traditional von Neumann architectures by eliminating multiple analog-to-digital conversions, data storage, and data movement operations. Current in-sensor processing approaches rely on tunable sensors or additional weighting elements to perform linear functions such as multiply accumulate operations as the sensor acquires data. This work implements in-sensor computing with an oscillatory retinal neuron device that converts incident optical signals into voltage oscillations. A computing scheme is introduced based on the frequency shift of coupled oscillators that enables parallel, frequency multiplexed, nonlinear operations on the inputs. An experimentally implemented 3 × 3 focal plane array of coupled neurons shows that functions approximating edge detection, thresholding, and segmentation occur in parallel. An example of inference on handwritten digits from the MNIST database is also experimentally demonstrated with a 3 × 3 array of coupled neurons feeding into a single hidden layer neural network, approximating a liquid-state machine. Finally, the equivalent energy consumption to carry out image processing operations, including peripherals such as the Fourier transform circuits, is projected to be <20 fJ/OP, possibly reaching as low as 15 aJ/OP.

Coupled Ferroelectric-Photonic Memory in a Retinomorphic Hardware for In-Sensor Computing.

The development of all-in-one devices for artificial visual systems offers an attractive solution in terms of energy efficiency and real-time processing speed. In recent years, the proliferation of smart sensors in the growth of Internet-of-Things (IoT) has led to the increasing importance of in-sensor computing technology, which places computational power at the edge of the data-flow architecture. In this study, a prototype visual sensor inspired by the human retina is proposed, which integrates ferroelectricity and photosensitivity in two-dimensional (2D) α-In2Se3 material. This device mimics the functions of photoreceptors and amacrine cells in the retina, performing optical reception and memory computation functions through the use of electrical switching polarization in the channel. The gate-tunable linearity of excitatory and inhibitory functions in photon-induced short-term plasticity enables to encode and classify 12 000 images in the Mixed National Institute of Standards and Technology (MNIST) dataset with remarkable accuracy, achieving ≈94%. Additionally, in-sensor convolution image processing through a network of phototransistors, with five convolutional kernels electrically pre-programmed into the transistors is demonstrated. The convoluted photocurrent matrices undergo straightforward arithmetic calculations to produce edge and feature-enhanced scenarios. The findings demonstrate the potential of ferroelectric α-In2Se3 for highly compact and efficient retinomorphic hardware implementation, regardless of ambipolar transport in the channel.

Optoelectronic Devices for In-Sensor Computing.

The demand for accurate perception of the physical world leads to a dramatic increase in sensory nodes. However, the transmission of massive and unstructured sensory data from sensors to computing units poses great challenges in terms of power-efficiency, transmission bandwidth, data storage, time latency, and security. To efficiently process massive sensory data, it is crucial to achieve data compression and structuring at the sensory terminals. In-sensor computing integrates perception, memory, and processing functions within sensors, enabling sensory terminals to perform data compression and data structuring. Here, vision sensors are adopted as an example and discuss the functions of electronic, optical, and optoelectronic hardware for visual processing. Particularly, hardware implementations of optoelectronic devices for in-sensor visual processing that can compress and structure multidimensional vision information are examined. The underlying resistive switching mechanisms of volatile/nonvolatile optoelectronic devices and their processing operations are explored. Finally, a perspective on the future development of optoelectronic devices for in-sensor computing is provided.

Gate-Tunable Positive and Negative Photoconductance in Near-Infrared Organic Heterostructures for In-Sensor Computing.

The rapid growth of sensor data in the artificial intelligence often causes significant reductions in processing speed and power efficiency. Addressing this challenge, in-sensor computing is introduced as an advanced sensor architecture that simultaneously senses, memorizes, and processes images at the sensor level. However, this is rarely reported for organic semiconductors that possess inherent flexibility and tunable bandgap. Herein, an organic heterostructure that exhibits a robust photoresponse to near-infrared (NIR) light is introduced, making it ideal for in-sensor computing applications. This heterostructure, consisting of partially overlapping p-type and n-type organic thin films, is compatible with conventional photolithography techniques, allowing for high integration density of up to 520 devices cm-2 with a 5 µm channel length. Importantly, by modulating gate voltage, both positive and negative photoresponses to NIR light (1050 nm) are attained, which establishes a linear correlation between responsivity and gate voltage and consequently enables real-time matrix multiplication within the sensor. As a result, this organic heterostructure facilitates efficient and precise NIR in-sensor computing, including image processing and nondestructive reading and classification, achieving a recognition accuracy of 97.06%. This work serves as a foundation for the development of reconfigurable and multifunctional NIR neuromorphic vision systems.

Retinomorphic Color Perception Based on Opponent Process Enabled by Perovskite Bipolar Photodetectors.

The ability to perceive color by the retina can be attributed to both its trichromatic photoreceptors and the antagonistic neural wiring known as the opponent process. While neuromorphic sensors have been shown to demonstrate memory and adaptation capabilities, color perception is still challenging due to the intrinsic lack of spectral selectivity in narrow bandgap semiconductors. Furthermore, research on emulating neural wiring is severely lacking. The combination of halide perovskite materials with a tunable bandgap and a novel bipolar photodetector design emulates the efficiency of the retina in processing color information. The stimuli-responsive material is also responsible for maintaining partial color constancy-an adaptation feature. Leveraging the unique enhancement of color contrasts, an in-sensor data compression and edge detection can also be demonstrated. The color perception, chromatic adaptation, and color contrast enhancement make perovskite bipolar photodetectors a unique example where the sensor and neural wiring can be co-developed in conjunction.

In-Sensor Tactile Fusion and Logic for Accurate Intention Recognition.

Touch control intention recognition is an important direction for the future development of human-machine interactions (HMIs). However, the implementation of parallel-sensing functional modules generally requires a combination of different logical blocks and control circuits, which results in regional redundancy, redundant data, and low efficiency. Here, a location-and-pressure intelligent tactile sensor (LPI tactile sensor) unprecedentedly combined with sensing, computing, and logic is proposed, enabling efficient and ultrahigh-resolution action-intention interaction. The LPI tactile sensor eliminates the need for data transfer among the functional units through the core integration design of the layered structure. It actuates in-sensor perception through feature transmission, fusion, and differentiation, thereby revolutionizing the traditional von Neumann architecture. While greatly simplifying the data dimensionality, the LPI tactile sensor achieves outstanding resolution sensing in both location (<400 µm) and pressure (75 Pa). Synchronous feature fusion and decoding support the high-fidelity recognition of action and combinatorial logic intentions. Benefiting from location and pressure synergy, the LPI tactile sensor demonstrates robust privacy as an encrypted password device and interaction intelligence through pressure enhancement. It can recognize continuous touch actions in real time, map real intentions to target events, and promote accurate and efficient intention-driven HMIs.

A Plasmonic Optoelectronic Resistive Random-Access Memory for In-Sensor Color Image Cryptography.

The optoelectronic resistive random-access memory (RRAM) with the integrated function of perception, storage and intrinsic randomness displays promising applications in the hardware level in-sensor image cryptography. In this work, 2D hexagonal boron nitride based optoelectronic RRAM is fabricated with semitransparent noble metal (Ag or Au) as top electrodes, which can simultaneous capture color image and generate physically unclonable function (PUF) key for in-sensor color image cryptography. Surface plasmons of noble metals enable the strong light absorption to realize an efficient modulation of filament growth at nanoscale. Resistive switching curves show that the optical stimuli can impede the filament aggregation and promote the filament annihilation, which originates from photothermal effects and photogenerated hot electrons in localized surface plasmon resonance of noble metals. By selecting noble metals, the optoelectronic RRAM array can respond to distinct wavelengths and mimic the biological dichromatic cone cells to perform the color perception. Due to the intrinsic and high-quality randomness, the optoelectronic RRAM can produce a PUF key in every exposure cycle, which can be applied in the reconfigurable cryptography. The findings demonstrate an effective strategy to build optoelectronic RRAM for in-sensor color image cryptography applications.

Emerging 2D Ferroelectric Devices for In-Sensor and In-Memory Computing.

The quantity of sensor nodes within current computing systems is rapidly increasing in tandem with the sensing data. The presence of a bottleneck in data transmission between the sensors, computing, and memory units obstructs the system's efficiency and speed. To minimize the latency of data transmission between units, novel in-memory and in-sensor computing architectures are proposed as alternatives to the conventional von Neumann architecture, aiming for data-intensive sensing and computing applications. The integration of 2D materials and 2D ferroelectric materials has been expected to build these novel sensing and computing architectures due to the dangling-bond-free surface, ultra-fast polarization flipping, and ultra-low power consumption of the 2D ferroelectrics. Here, the recent progress of 2D ferroelectric devices for in-sensing and in-memory neuromorphic computing is reviewed. Experimental and theoretical progresses on 2D ferroelectric devices, including passive ferroelectrics-integrated 2D devices and active ferroelectrics-integrated 2D devices, are reviewed followed by the integration of perception, memory, and computing application. Notably, 2D ferroelectric devices have been used to simulate synaptic weights, neuronal model functions, and neural networks for image processing. As an emerging device configuration, 2D ferroelectric devices have the potential to expand into the sensor-memory and computing integration application field, leading to new possibilities for modern electronics.

One-Phototransistor-One-Memristor Array with High-Linearity Light-Tunable Weight for Optic Neuromorphic Computing.

The recent advances in optic neuromorphic devices have led to a subsequent rise in use for construction of energy-efficient artificial-vision systems. The widespread use can be attributed to their ability to capture, store, and process visual information from the environment. The primary limitations of existing optic neuromorphic devices include nonlinear weight updates, cross-talk issues, and silicon process incompatibility. In this study, a highly linear, light-tunable, cross-talk-free, and silicon-compatible one-phototransistor-one-memristor (1PT1R) optic memristor is experimentally demonstrated for the implementation of an optic artificial neural network (OANN). For optic image recognition in the experiment, an OANN is constructed using a 16 × 3 1PT1R memristor array, and it is trained on an online platform. The model yields an accuracy of 99.3% after only ten training epochs. The 1PT1R memristor, which shows good performance, demonstrates its ability as an excellent hardware solution for highly efficient optic neuromorphic and edge computing.

In-Sensor Computing Realization Using Fully CMOS-Compatible TiN/HfOx-Based Neuristor Array.

With the evolution of artificial intelligence, the explosive growth of data from sensory terminals gives rise to severe energy-efficiency bottleneck issues due to cumbersome data interactions among sensory, memory, and computing modules. Heterogeneous integration methods such as chiplet technology can significantly reduce unnecessary data movement; however, they fail to address the fundamental issue of the substantial time and energy overheads resulting from the physical separation of computing and sensory components. Brain-inspired in-sensor neuromorphic computing (ISNC) has plenty of room for such data-intensive applications. However, one key obstacle in developing ISNC systems is the lack of compatibility between material systems and manufacturing processes deployed in sensors and computing units. This study successfully addresses this challenge by implementing fully CMOS-compatible TiN/HfOx-based neuristor array. The developed ISNC system demonstrates several advantageous features, including multilevel analogue modulation, minimal dispersion, and no significant degradation in conductance (@125 °C). These characteristics enable stable and reproducible neuromorphic computing. Additionally, the device exhibits modulatable sensory and multi-store memory processes. Furthermore, the system achieves information recognition with a high accuracy rate of 93%, along with frequency selectivity and notable activity-dependent plasticity. This work provides a promising route to affordable and highly efficient sensory neuromorphic systems.

Artificial Visual Systems Fabricated with Ferroelectric van der Waals Heterostructure for In-Memory Computing Applications.

Rapid developments in artificial neural network techniques and retina-inspired artificial visual systems are required to realize the sensing, processing, and memorization of an optical signal in a single device. Herein, a ferroelectric field-effect transistor fabricated with CuInP2S6 and α-In2Se3 van der Waals heterostructures is proposed and demonstrated for the development of an artificial visual system. The dipole polarizations are coupled and bidirectionally locked inside the ferroelectric α-In2Se3 along the in-plane and out-of-plane directions and are controlled by the gate voltages. Furthermore, light-induced polarization can change the order of polarization of the dipoles inside α-In2Se3. We demonstrate that using the combined control of these electrical and optical signals, the device may function like a retina-inspired vision system. The device can operate across a wide wavelength range (405-850 nm) and detect very low incident light (0.03 mW/cm2). Color recognition, high paired-pulse facilitation (∼170%), and short- to long-term memory transitions through quick learning are observed using this device. Additionally, this device demonstrates different complex processing abilities, including pattern recognition, light adaptation, optical logic operation, and event learning. The proposed ferroelectric heterostructure-based artificial visual system can serve as an essential bridge for fulfilling the future requirements of all-in-one sensing and memory-processing devices.

A Reconfigurable Optoelectronic Synaptic Transistor with Stable Zr-CsPbI3 Nanocrystals for Visuomorphic Computing.

Reconfigurable phototransistor memory attracts considerable attention for adaptive visuomorphic computing, with highly efficient sensing, memory, and processing functions integrated onto a single device. However, developing reconfigurable phototransistor memory remains a challenge due to the lack of an all-optically controlled transition between short-term plasticity (STP) and long-term plasticity (LTP). Herein, an air-stable Zr-CsPbI3 perovskite nanocrystal (PNC)-based phototransistor memory is designed, which is capable of broadband photoresponses. Benefitting from the different electron capture ability of Zr-CsPbI3 PNCs to 650 and 405 nm light, an artificial synapse and non-volatile memory can be created on-demand and quickly reconfigured within a single device for specific purposes. Owing to the optically reconfigurable and wavelength-aware operation between STP and LTP modes, the integrated blue feature extraction and target recognition can be demonstrated in a homogeneous neuromorphic vision sensor array. This work suggests a new way in developing perovskite optoelectronic transistors for highly efficient in-sensor computing.