Optimizing Image Enhancement: Feature Engineering for Improved Classification in AI-Assisted Artificial Retinas.

Artificial retinas have revolutionized the lives of many blind people by enabling their ability to perceive vision via an implanted chip. Despite significant advancements, there are some limitations that cannot be ignored. Presenting all objects captured in a scene makes their identification difficult. Addressing this limitation is necessary because the artificial retina can utilize a very limited number of pixels to represent vision information. This problem in a multi-object scenario can be mitigated by enhancing images such that only the major objects are considered to be shown in vision. Although simple techniques like edge detection are used, they fall short in representing identifiable objects in complex scenarios, suggesting the idea of integrating primary object edges. To support this idea, the proposed classification model aims at identifying the primary objects based on a suggested set of selective features. The proposed classification model can then be equipped into the artificial retina system for filtering multiple primary objects to enhance vision. The suitability of handling multi-objects enables the system to cope with real-world complex scenarios. The proposed classification model is based on a multi-label deep neural network, specifically designed to leverage from the selective feature set. Initially, the enhanced images proposed in this research are compared with the ones that utilize an edge detection technique for single, dual, and multi-object images. These enhancements are also verified through an intensity profile analysis. Subsequently, the proposed classification model's performance is evaluated to show the significance of utilizing the suggested features. This includes evaluating the model's ability to correctly classify the top five, four, three, two, and one object(s), with respective accuracies of up to 84.8%, 85.2%, 86.8%, 91.8%, and 96.4%. Several comparisons such as training/validation loss and accuracies, precision, recall, specificity, and area under a curve indicate reliable results. Based on the overall evaluation of this study, it is concluded that using the suggested set of selective features not only improves the classification model's performance, but aligns with the specific problem to address the challenge of correctly identifying objects in multi-object scenarios. Therefore, the proposed classification model designed on the basis of selective features is considered to be a very useful tool in supporting the idea of optimizing image enhancement.

Integrated In-Sensor Computing Optoelectronic Device for Environment-Adaptable Artificial Retina Perception Application.

With the development and application of artificial intelligence, there is an appeal to the exploitation of various sensors and memories. As the most important perception of human beings, vision occupies more than 80% of all the received information. Inspired by biological eyes, an artificial retina based on 2D Janus MoSSe was fabricated, which could simulate functions of visual perception with electronic/ion and optical comodulation. Furthermore, inspired by human brain, sensing, memory, and neuromorphic computing functions were integrated on one device for multifunctional intelligent electronics, which was beneficial for scalability and high efficiency. Through the formation of faradic electric double layer (EDL) at the metal-oxide/electrolyte interfaces could realize synaptic weight changes. On the basis of the optoelectronic performances, light adaptation of biological eyes, preprocessing, and recognition of handwritten digits were implemented successfully. This work may provide a strategy for the future integrated sensing-memory-processing device for optoelectronic artificial retina perception application.

[Modern concepts of bionic vision]. / Sovremennye kontseptsii bionicheskogo zreniya.

Loss of vision is a pressing medical and social problem leading to profound disability, loss of ability to work, serious alterations in the psycho-emotional state, and a decline of the quality of life. When conservative or surgical treatment can not help restore vision, the use of visual prosthesis - bionic eye - can be an effective solution. This review covers the main modern approaches to the development of visual prosthetic systems. Analysis of publications revealed that there are several main approaches to visual prosthesis differing primarily by the anatomical structure targeted for stimulation in order to activate visual sensations. The most significant among them are retinal prostheses, optic nerve stimulation, and cortical visual prostheses. Currently, retinal prostheses such as ARGUS II demonstrate the most successful results, since the stimulation of the surviving neural structures of the retina is a relatively easy task, but their field of application is limited to diseases associated with pathological changes in photoreceptors. The development of cortical visual prostheses is more difficult, but in the future they may allow using more stimulation channels to obtain a more detailed visual perception. In addition, cortical visual prostheses are universal, as they do not require preservation of any structures of the visual organ, only the primary visual cortex.

The use of Argus® II retinal prosthesis by blind subjects to achieve localisation and prehension of objects in 3-dimensional space.

BACKGROUND: The Argus® II retinal prosthesis system has entered mainstream treatment for patients blind from Retinitis Pigmentosa (RP). We set out to evaluate the use of this system by blind subjects to achieve object localisation and prehension in 3-dimensional space. METHODS: This is a single-centre, prospective, internally-controlled case series involving 5 blind RP subjects who received the Argus® II implant. The subjects were instructed to visually locate, reach and grasp (i.e. prehension) a small white cuboid object placed at random locations on a black worktop. A flashing LED beacon was attached to the reaching index finger (as a finger marker) to assess the effect of enhanced finger visualisation on performance. Tasks were performed with the prosthesis switched "on" or "off" and with the finger marker switched "on" or "off". Forty-eight trials were performed per subject. Trajectory of each subject's hand movement during the task was recorded by a 3D motion-capture unit (Qualysis®, see supplementary video) and analysed using a MATLAB script. RESULT: Percentage of successful prehension±standard deviation was: 71.3 ± 27.1 % with prosthesis on and finger marker on; 77.5 ± 24.5 % with prosthesis on and finger marker off; 0.0 ± 0.0 % with prosthesis off and finger marker on, and 0.00 ± 0.00 % with prosthesis off and finger marker off. The finger marker did not have a significant effect on performance (P = 0.546 and 1, Wilcoxon Signed Rank test, with prosthesis on and off respectively). With prosthesis off, none of the subjects were able to visually locate the target object and no initiation of prehension was attempted. With prosthesis on, prehension was initiated on 82.5 % (range 59-100 %) of the trials with 89.0 % (range 66.7-100 %) achieving successful prehension. CONCLUSION: Argus® II subjects were able to achieve object localisation and prehension better with their prosthesis switched on than off.

A review and update on the current status of retinal prostheses (bionic eye).

INTRODUCTION/BACKGROUND: The Argus® II is the first retinal prosthesis approved for the treatment of patients blind from retinitis pigmentosa (RP), receiving CE (Conformité Européenne) marking in March 2011 and FDA approval in February 2013. Alpha-IMS followed closely and obtained CE marking in July 2013. Other devices are being developed, some of which are currently in clinical trials. SOURCES OF DATA: A systematic literature search was conducted on PubMED, Google Scholar and IEEExplore. AREAS OF AGREEMENT: Retinal prostheses play a part in restoring vision in blind RP patients providing stable, safe and long-term retinal stimulation. AREAS OF CONTROVERSY: Objective improvement in visual function does not always translate into consistent improvement in the patient's quality of life. Controversy exists over the use of an external image-capturing device versus internally placed photodiode devices. GROWING POINTS: The alpha-IMS, a photovoltaic-based retinal prosthesis recently obtained its CE marking in July 2013. AREAS TIMELY FOR DEVELOPING RESEARCH: Improvement in retinal prosthetic vision depends on: (i) improving visual resolution, (ii) improving the visual field, (iii) developing an accurate neural code for image processing and (iv) improving the biocompatibility of the device to ensure longevity.

Wireless and Opto-Stimulated Flexible Implants: Artificial Retina Constructed by Ferroelectric BiFeO3-BaTiO3/P(VDF-TrFE) Composites.

The degeneration of retinal photoreceptors is one of the primary causes of blindness, and the implantation of retinal prostheses offers hope for vision restoration in individuals who are completely blind. Flexible bioelectronic devices present a promising avenue for the next generation of retinal prostheses owing to their soft mechanical properties and tissue friendliness. In this study, we developed flexible composite films of ferroelectric BiFeO3-BaTiO3 (BFO-BTO) particles synthesized by the hydrothermal method and ferroelectric poly(vinyldene difluoride-trifluoroethylene) (P(VDF-TrFE)) polymer and investigated their applications in artificial retinas. Owing to the coupling of the photothermal effect of BFO-BTO particles and the pyroelectric effect of the P(VDF-TrFE) polymer, the composite films demonstrate a strong photoelectric response (a maximum peak-to-peak photovoltage > 80 V under blue light of 100 mW/cm2) in a wide wavelength range of light (from visible to infrared) with the inherent flexibility and ease of preparation, making it an attractive candidate for artificial retinal applications. Experimental results showed that blind rats implanted with artificial retinas of the composites display light-responsive behavior, showcasing the effectiveness of vision restoration. This study demonstrates a novel approach for employing ferroelectric materials in vision restoration and offers insights into future artificial retina design.

Artificial Amacrine Retinal Circuits.

Event-based imaging represents a new paradigm in visual information processing that addresses the speed and energy efficiency shortcomings inherently present in the current complementary metal oxide semiconductor-based machine vision. Realizing such imaging systems has previously been sought using very large-scale integration technologies that have complex circuitries consisting of many photodiodes, differential amplifiers, capacitors, and resistors. Here, we demonstrate that event-driven sensing can be achieved using a simple one-resistor, one-capacitor (1R1C) circuit, where the capacitor is modified with colloidal quantum dots (CQDs) to have a photoresponse. This sensory circuit emulates the motion-tracking function of the biological retina, in which the amacrine cells in the bipolar-to-ganglion synaptic pathway produce a transient spiking signal only in response to changes in light intensity but remain inactive under constant illumination. When extended to a 2D imaging array, the individual sensors work independently and output signals only when a change in the light intensity is detected; hence, the concept of the frame in image processing is thereby removed. In this work, we present the fabrication and characterization of a CQD photocapacitor-based 1R1C circuit that has a spectral response at 1550 nm in the short-wave infrared (SWIR). We report on the key performance parameters including peak responsivity, noise, and optical noise equivalent power and discuss the operating mechanism that is responsible for spiking responses in these artificial retinal circuits. The present work sets the foundation for expanding the bioinspired vision sensor capability toward midwave infrared (MWIR) and long-wave infrared (LWIR) spectral regions that are invisible to human eyes and mainstream semiconductor technologies.

Design and simulation of artificial retinal stimulation IC with switched capacitor using Si nanowire optical properties.

This study introduces an approach for converting the current from a sensor into controllable voltage. To this end, a switched-capacitor structure was integrated to provide efficient current-to-voltage conversion. The generated voltage was further regulated by an operational amplifier current source, enhancing stability and precision. An n-type metal oxide semiconductor field-effect transistor structure under an H-bridge was integrated into the system to achieve fine-tuned control over current stimulation. This component contributed to voltage regulation and enabled bi-directional control of current flow, offering versatility in adjusting current amplitudes using working and counter electrodes. This dynamic control mechanism was pivotal for effectively controlling the intensity of current stimulation. We applied Verilog-A modeling to simulate the optical characteristics of Si nanowires. The proposed system efficiently converted sensor-derived current into voltage using a switched-capacitor structure. Simultaneously, the precision was enhanced via operational amplifier regulation and n-type metal-oxide-semiconductor field-effect transistor-based H-bridge control. The simulation showed a current stimulus amplitude ranging from 2 to 13 µA for a variable photocurrent of Si nanowires (Rex: 10 kΩ, pulse: 100 Hz, 1 ms). The ability to finely control current stimulation intensity holds promise for diverse applications requiring accurate and adjustable current manipulation. This study contributes to the growing field of sensor technology by offering a unique perspective on the integration of nanostructures and electronic components for an enhanced control and functionality.

Tunable Negative and Positive Photoconductance in Van Der Waals Heterostructure for Image Preprocessing.

The processing of visual information occurs mainly in the retina, and the retinal preprocessing function greatly improves the transmission quality and efficiency of visual information. The artificial retina system provides a promising path to efficient image processing. Here, graphene/InSe/h-BN heterogeneous structure is proposed, which exhibits negative and positive photoconductance (NPC and PPC) effects by altering the strength of a single wavelength laser. Moreover, a modified theoretical model is presented based on the power-dependent photoconductivity effect of laser: I ph = - mP α 1 + nP α 2 ${\rm I}_{\rm ph}\,=\,-{\rm mP}^{\alpha _{1}} + {\rm nP}^{\alpha _{2}}$ , which can reveal the internal physical mechanism of negative/positive photoconductance effects. The present 2D structure design allows the field effect transistor (FET) to exhibit excellent photoelectric performance (RNPC = 1.1× 104 AW-1, RPPC = 13 AW-1) and performance stability. Especially, the retinal pretreatment process is successfully simulated based on the negative and positive photoconductive effects. Moreover, the pulse signal input improves the device responsivity by 167%, and the transmission quality and efficiency of the visual signal can also be enhanced. This work provides a new design idea and direction for the construction of artificial vision, and lay a foundation for the integration of the next generation of optoelectronic devices.

2D Semiconductor Based Flexible Photoresponsive Ring Oscillators for Artificial Vision Pixels.

Artificial retina implantation provides an effective and feasible attempt for vision recovery in addition to retinal transplantation. The most advanced artificial retinas ever developed based on silicon technology are rigid and thus less compatible with the biosystem. Here we demonstrate flexible photoresponsive ring oscillators (PROs) based on the 2D semiconductor MoS2 for artificial retinas. Under natural light illuminations, arrayed PROs on flexible substrates serving as vision pixels can efficiently output light-intensity-dependent electrical pulses that are processable and transmittable in the human visual nerve system. Such PROs can work under low supply voltages below 1 V with a record-low power consumption, e.g. only 12.4 nW at a light intensity of 10 mW/cm2, decreased by ∼500 times compared with that of the state-of-the-art silicon devices. Such flexible artificial retinas with a simple device structure, high light-to-signal conversion efficiency, ultralow power consumption, and high tunability provide an alternative prosthesis for further clinical trials.

Applications of Graphene in Five Senses, Nervous System, and Artificial Muscles.

Graphene remains of great interest in biomedical applications because of biocompatibility. Diseases relating to human senses interfere with life satisfaction and happiness. Therefore, the restoration by artificial organs or sensory devices may bring a bright future by the recovery of senses in patients. In this review, we update the most recent progress in graphene based sensors for mimicking human senses such as artificial retina for image sensors, artificial eardrums, gas sensors, chemical sensors, and tactile sensors. The brain-like processors are discussed based on conventional transistors as well as memristor related neuromorphic computing. The brain-machine interface is introduced for providing a single pathway. Besides, the artificial muscles based on graphene are summarized in the means of actuators in order to react to the physical world. Future opportunities remain for elevating the performances of human-like sensors and their clinical applications.

Neurostimulators for high-resolution artificial retina: ASIC design challenges and solutions.

Objective.Neurostimulator is one of the most important part in artificial retina design. In this paper, we discuss the main challenges in the design of application-specific integrated circuit for high-resolution artificial retina and suggest corresponding solutions.Approach. Problems in the design of the neurostimulator for the existing artificial retina have not been solved yet are analyzed and solutions are presented. For verification of the solutions, mathematical proof, MATLAB and Ansys simulations are used.Main results. The drawbacks of resorting to a high-voltage complementary metal oxide semiconductor (CMOS) process to deal with the large voltage compliance demanded by the stimulator output stage are pointed out, and an alternative approach based on a circuit that switches the voltage of the common reference electrode is proposed to overcome. The necessity of an active discharge circuit to remove the residual charge of electrodes caused by an unbalanced stimulus is investigated. We present a circuit analysis showing that the use of a passive discharge circuit is sufficient to suppress problematic direct current in most situations. Finally, possible restrictions on input and output (I/O) count are investigated by estimating the resistive-capacitive delay caused by the interconnection between the I/O pad and the microelectrode array.Significance. The results of this paper clarified the problems currently faced by neurostimulator design for the artificial retina. Through the solutions presented in this study, circuits with more competitiveness in power and area consumption can be designed.

Preferential modulation of individual retinal ganglion cells by electrical stimulation.

Objective.Retinal prostheses have had limited success in vision restoration through electrical stimulation of surviving retinal ganglion cells (RGCs) in the degenerated retina. This is partly due to non-preferential stimulation of all RGCs near a single stimulating electrode, which include cells that conflict in their response properties and their contribution to visiual processing. Our study proposes a stimulation strategy to preferentially stimulate individual RGCs based on their temporal electrical receptive fields (tERFs).Approach.We recorded the responses of RGCs using whole-cell patch clamping and demonstrated the stimulation strategy, first using intracellular stimulation, then via extracellular stimulation.Main results. We successfully reconstructed the tERFs according to the RGC response to Gaussian white noise current stimulation. The characteristics of the tERFs were extracted and compared based on the morphological and light response types of the cells. By re-delivering stimulation trains that were composed of the tERFs obtained from different cells, we could preferentially stimulate individual RGCs as the cells showed lower activation thresholds to their own tERFs.Significance.This proposed stimulation strategy implemented in the next generation of recording and stimulating retinal prostheses may improve the quality of artificial vision.

Individual variability of neural computations in the primate retina.

Variation in the neural code contributes to making each individual unique. We probed neural code variation using ∼100 population recordings from major ganglion cell types in the macaque retina, combined with an interpretable computational representation of individual variability. This representation captured variation and covariation in properties such as nonlinearity, temporal dynamics, and spatial receptive field size and preserved invariances such as asymmetries between On and Off cells. The covariation of response properties in different cell types was associated with the proximity of lamination of their synaptic input. Surprisingly, male retinas exhibited higher firing rates and faster temporal integration than female retinas. Exploiting data from previously recorded retinas enabled efficient characterization of a new macaque retina, and of a human retina. Simulations indicated that combining a large dataset of retinal recordings with behavioral feedback could reveal the neural code in a living human and thus improve vision restoration with retinal implants.

Conception of a Smart Artificial Retina Based on a Dual-Mode Organic Sensing Inverter.

The human visual system enables perceiving, learning, remembering, and recognizing elementary visual information (light, colors, and images), which has inspired the development of biomimicry visual system-based electronic devices. Photosensing and synaptic devices are integrated into these systems to realize elementary information storage and recognition to imitate image processing. However, the severe restrictions of the monotonic light response and complicated circuitry design remain challenges for the development of artificial visual devices. Here, the concept of a smart artificial retina based on an organic optical sensing inverter device that can be operated as a multiwavelength photodetector and recorder is reported first. The device exhibits a light-triggered broadband (red/green/blue) response, a low energy consumption as low as ±5 V, and an ultrafast response speed (<300 ms). Moreover, the multifunctional component is also combined within a single cell for health monitoring of the artificial retina during light surveillance to avoid retinopathy. Proof-of-concept devices, by simplifying the circuitry and providing dual-mode functions, can contribute significantly to the development of bionics design and broaden the horizon for smart artificial retinas in the human visual system.

Event-Based Sensing and Signal Processing in the Visual, Auditory, and Olfactory Domain: A Review.

The nervous systems converts the physical quantities sensed by its primary receptors into trains of events that are then processed in the brain. The unmatched efficiency in information processing has long inspired engineers to seek brain-like approaches to sensing and signal processing. The key principle pursued in neuromorphic sensing is to shed the traditional approach of periodic sampling in favor of an event-driven scheme that mimicks sampling as it occurs in the nervous system, where events are preferably emitted upon the change of the sensed stimulus. In this paper we highlight the advantages and challenges of event-based sensing and signal processing in the visual, auditory and olfactory domains. We also provide a survey of the literature covering neuromorphic sensing and signal processing in all three modalities. Our aim is to facilitate research in event-based sensing and signal processing by providing a comprehensive overview of the research performed previously as well as highlighting conceptual advantages, current progress and future challenges in the field.

Folate-Modified Photoelectric Responsive Polymer Microarray as Bionic Artificial Retina to Restore Visual Function.

A high-optical-resolution artificial retina system that accurately communicates with the optic nerve is the main challenge in the modern biological science and bionic field. Here, we developed a bionic artificial retina possessing phototransduction "cells" with measurements even smaller than that of the neural cells. Using the technique of micrometer processing, we constructed a pyramid-shape periodic microarray of a photoreceptor. Each "sensing cell" took advantage of polythiophene derivative/fullerene derivative (PCBM) as a photoelectric converter. Because folic acid played an essential role in eye growth, we particularly modified the polythiophene derivatives with folic acid tags. Therefore, the artificial retina could enlarge the contact area and even recognize the nerve cells to improve the consequence of nerve stimulation. We implanted the artificial retina into blinded rats' eyes. Electrophysiological analysis revealed its recovery of photosensitive function 3 months after surgery. Our work provides an innovative idea for fabricating a high-resolution bionic artificial retina system. It shows great potential in artificial intelligence and biomedicine.

Retina-Inspired Carbon Nitride-Based Photonic Synapses for Selective Detection of UV Light.

Photonic synapses combine sensing and processing in a single device, so they are promising candidates to emulate visual perception of a biological retina. However, photonic synapses with wavelength selectivity, which is a key property for visual perception, have not been developed so far. Herein, organic photonic synapses that selectively detect UV rays and process various optical stimuli are presented. The photonic synapses use carbon nitride (C3 N4 ) as an UV-responsive floating-gate layer in transistor geometry. C3 N4 nanodots dominantly absorb UV light; this trait is the basis of UV selectivity in these photonic synapses. The presented devices consume only 18.06 fJ per synaptic event, which is comparable to the energy consumption of biological synapses. Furthermore, in situ modulation of exposure to UV light is demonstrated by integrating the devices with UV transmittance modulators. These smart systems can be further developed to combine detection and dose-calculation to determine how and when to decrease UV transmittance for preventive health care.

Recent Advances in Vertically Aligned Nanowires for Photonics Applications.

Over the past few decades, nanowires have arisen as a centerpiece in various fields of application from electronics to photonics, and, recently, even in bio-devices. Vertically aligned nanowires are a particularly decent example of commercially manufacturable nanostructures with regard to its packing fraction and matured fabrication techniques, which is promising for mass-production and low fabrication cost. Here, we track recent advances in vertically aligned nanowires focused in the area of photonics applications. Begin with the core optical properties in nanowires, this review mainly highlights the photonics applications such as light-emitting diodes, lasers, spectral filters, structural coloration and artificial retina using vertically aligned nanowires with the essential fabrication methods based on top-down and bottom-up approaches. Finally, the remaining challenges will be briefly discussed to provide future directions.

CVD growth of perovskite/graphene films for high-performance flexible image sensor.

Hybrid perovskite possesses excellent photoelectric properties, including large light-absorption capacity and high carrier mobility, and is an ideal light-absorbing material for photoelectric devices. The grain size and compactness of hybrid perovskite are key factors affecting the performance of photoelectric devices. The photocurrent and photoresponsivity of these devices are relatively low because of the rapidly recombined photoexcited electron-hole pairs in hybrid perovskite. Herein, we develop a facile two-step chemical vapor deposition (CVD) method to synthesize a high-quality van der Waals (vdWs) MAPbI3/graphene heterostructure for high-performance image sensor. We introduced inorganic sources (PbI2) to vdWs epitaxially grown PbI2 film on a seamless graphene monolayer film template through CVD. Methylammonium iodide (MAI) was then reintroduced to prepare the vdWs MAPbI3/graphene heterostructure. The MAPbI3 layer is composed of densely packed, large-size grains and displays a smooth surface. High photoresponsivity of 107 A/W is achieved in the corresponding photodetector. Inspired by the human visual system, we designed a flexible photodetector array containing (24 × 24) pixels, achieving perfect image recognition and color discrimination. Our study may greatly facilitate the construction of high-performance optoelectronic devices in artificial retina, biomedical imaging, remote sensing, and optical communication.