Isolated single sound lip-reading using a frame-based camera and event-based camera.

Unlike the conventional frame-based camera, the event-based camera detects changes in the brightness value for each pixel over time. This research work on lip-reading as a new application by the event-based camera. This paper proposes an event camera-based lip-reading for isolated single sound recognition. The proposed method consists of imaging from event data, face and facial feature points detection, and recognition using a Temporal Convolutional Network. Furthermore, this paper proposes a method that combines the two modalities of the frame-based camera and an event-based camera. In order to evaluate the proposed method, the utterance scenes of 15 Japanese consonants from 20 speakers were collected using an event-based camera and a video camera and constructed an original dataset. Several experiments were conducted by generating images at multiple frame rates from an event-based camera. As a result, the highest recognition accuracy was obtained in the image of the event-based camera at 60 fps. Moreover, it was confirmed that combining two modalities yields higher recognition accuracy than a single modality.

Stepwise Observation of Iodine Diffusion in a Flexible Coordination Network Having Dual Interactive Sites.

We created dual interactive sites in a porous coordination network using a CuI cluster and a rotation-restricted ligand, tetra(3-pyridyl)phenylmethane (3-TPPM). The dual interactive sites of iodide and Cu ions can adsorb I2 via four-step processes including two chemisorption processes. Initially, one I2 molecule was physisorbed in a pore and successively chemisorbed on iodide sites of the pore surface, and then the next I2 molecule was physisorbed and chemisorbed on Cu ions to form a cross-linked network. We revealed the four-step I2 diffusion process by single-crystal X-ray structure determination and spectroscopic kinetic analysis.