Bioinspired In-Sensor Multimodal Fusion for Enhanced Spatial and Spatiotemporal Association.

Multimodal perception can capture more precise and comprehensive information compared with unimodal approaches. However, current sensory systems typically merge multimodal signals at computing terminals following parallel processing and transmission, which results in the potential loss of spatial association information and requires time stamps to maintain temporal coherence for time-series data. Here we demonstrate bioinspired in-sensor multimodal fusion, which effectively enhances comprehensive perception and reduces the level of data transfer between sensory terminal and computation units. By adopting floating gate phototransistors with reconfigurable photoresponse plasticity, we realize the agile spatial and spatiotemporal fusion under nonvolatile and volatile photoresponse modes. To realize an optimal spatial estimation, we integrate spatial information from visual-tactile signals. For dynamic events, we capture and fuse in real time spatiotemporal information from visual-audio signals, realizing a dance-music synchronization recognition task without a time-stamping process. This in-sensor multimodal fusion approach provides the potential to simplify the multimodal integration system, extending the in-sensor computing paradigm.

Multidimensional vision sensors for information processing.

The visual scene in the physical world integrates multidimensional information (spatial, temporal, polarization, spectrum and so on) and typically shows unstructured characteristics. Conventional image sensors cannot process this multidimensional vision data, creating a need for vision sensors that can efficiently extract features from substantial multidimensional vision data. Vision sensors are able to transform the unstructured visual scene into featured information without relying on sophisticated algorithms and complex hardware. The response characteristics of sensors can be abstracted into operators with specific functionalities, allowing for the efficient processing of perceptual information. In this Review, we delve into the hardware implementation of multidimensional vision sensors, exploring their working mechanisms and design principles. We exemplify multidimensional vision sensors built on emerging devices and silicon-based system integration. We further provide benchmarking metrics for multidimensional vision sensors and conclude with the principle of device-system co-design and co-optimization.

Optoelectronic Devices for In-Sensor Computing.

The demand for accurate perception of the physical world leads to a dramatic increase in sensory nodes. However, the transmission of massive and unstructured sensory data from sensors to computing units poses great challenges in terms of power-efficiency, transmission bandwidth, data storage, time latency, and security. To efficiently process massive sensory data, it is crucial to achieve data compression and structuring at the sensory terminals. In-sensor computing integrates perception, memory, and processing functions within sensors, enabling sensory terminals to perform data compression and data structuring. Here, vision sensors are adopted as an example and discuss the functions of electronic, optical, and optoelectronic hardware for visual processing. Particularly, hardware implementations of optoelectronic devices for in-sensor visual processing that can compress and structure multidimensional vision information are examined. The underlying resistive switching mechanisms of volatile/nonvolatile optoelectronic devices and their processing operations are explored. Finally, a perspective on the future development of optoelectronic devices for in-sensor computing is provided.

In-Sensor Computing: Materials, Devices, and Integration Technologies.

The number of sensor nodes in the Internet of Things is growing rapidly, leading to a large volume of data generated at sensory terminals. Frequent data transfer between the sensors and computing units causes severe limitations on the system performance in terms of energy efficiency, speed, and security. To efficiently process a substantial amount of sensory data, a novel computation paradigm that can integrate computing functions into sensor networks should be developed. The in-sensor computing paradigm reduces data transfer and also decreases the high computing complexity by processing data locally. Here, the hardware implementation of the in-sensor computing paradigm at the device and array levels is discussed. The physical mechanisms that lead to unique sensory response characteristics and their corresponding computing functions are illustrated. In particular, bioinspired device characteristics enable the implementation of the functionalities of neuromorphic computation. The integration technology is also discussed and the perspective on the future development of in-sensor computing is provided.

Optoelectronic graded neurons for bioinspired in-sensor motion perception.

Motion processing has proven to be a computational challenge and demands considerable computational resources. Contrast this with the fact that flying insects can agilely perceive real-world motion with their tiny vision system. Here we show that phototransistor arrays can directly perceive different types of motion at sensory terminals, emulating the non-spiking graded neurons of insect vision systems. The charge dynamics of the shallow trapping centres in MoS2 phototransistors mimic the characteristics of graded neurons, showing an information transmission rate of 1,200 bit s-1 and effectively encoding temporal light information. We used a 20 × 20 photosensor array to detect trajectories in the visual field, allowing the efficient perception of the direction and vision saliency of moving objects and achieving 99.2% recognition accuracy with a four-layer neural network. By modulating the charge dynamics of the shallow trapping centres of MoS2, the sensor array can recognize motion with a temporal resolution ranging from 101 to 106 ms.

Highly Trustworthy In-Sensor Cryptography for Image Encryption and Authentication.

The prevailing transmission of image information over the Internet of Things demands trustworthy cryptography for high security and privacy. State-of-the-art security modules are usually physically separated from the sensory terminals that capture images, which unavoidably exposes image information to various attacks during the transmission process. Here we develop in-sensor cryptography that enables capturing images and producing security keys in the same hardware devices. The generated key inherently binds to the captured images, which gives rise to highly trustworthy cryptography. Using the intrinsic electronic and optoelectronic characteristics of the 256 molybdenum disulfide phototransistor array, we can harvest electronic and optoelectronic binary keys with a physically unclonable function and further upgrade them into multiple-state ternary and double-binary keys, exhibiting high uniformity, uniqueness, randomness, and coding capacity. This in-sensor cryptography enables highly trustworthy image encryption to avoid passive attacks and image authentication to prevent unauthorized editions.

Room-temperature valley transistors for low-power neuromorphic computing.

Valley pseudospin is an electronic degree of freedom that promises highly efficient information processing applications. However, valley-polarized excitons usually have short pico-second lifetimes, which limits the room-temperature applicability of valleytronic devices. Here, we demonstrate room-temperature valley transistors that operate by generating free carrier valley polarization with a long lifetime. This is achieved by electrostatic manipulation of the non-trivial band topology of the Weyl semiconductor tellurium (Te). We observe valley-polarized diffusion lengths of more than 7 µm and fabricate valley transistors with an ON/OFF ratio of 105 at room temperature. Moreover, we demonstrate an ion insertion/extraction device structure that enables 32 non-volatile memory states with high linearity and symmetry in the Te valley transistor. With ultralow power consumption (~fW valley contribution), we enable the inferring process of artificial neural networks, exhibiting potential for applications in low-power neuromorphic computing.

Standards for the Characterization of Endurance in Resistive Switching Devices.

Resistive switching (RS) devices are emerging electronic components that could have applications in multiple types of integrated circuits, including electronic memories, true random number generators, radiofrequency switches, neuromorphic vision sensors, and artificial neural networks. The main factor hindering the massive employment of RS devices in commercial circuits is related to variability and reliability issues, which are usually evaluated through switching endurance tests. However, we note that most studies that claimed high endurances >106 cycles were based on resistance versus cycle plots that contain very few data points (in many cases even <20), and which are collected in only one device. We recommend not to use such a characterization method because it is highly inaccurate and unreliable (i.e., it cannot reliably demonstrate that the device effectively switches in every cycle and it ignores cycle-to-cycle and device-to-device variability). This has created a blurry vision of the real performance of RS devices and in many cases has exaggerated their potential. This article proposes and describes a method for the correct characterization of switching endurance in RS devices; this method aims to construct endurance plots showing one data point per cycle and resistive state and combine data from multiple devices. Adopting this recommended method should result in more reliable literature in the field of RS technologies, which should accelerate their integration in commercial products.