Understanding Human Cognition Through Computational Modeling.

One important goal of cognitive science is to understand the mind in terms of its representational and computational capacities, where computational modeling plays an essential role in providing theoretical explanations and predictions of human behavior and mental phenomena. In my research, I have been using computational modeling, together with behavioral experiments and cognitive neuroscience methods, to investigate the information processing mechanisms underlying learning and visual cognition in terms of perceptual representation and attention strategy. In perceptual representation, I have used neural network models to understand how the split architecture in the human visual system influences visual cognition, and to examine perceptual representation development as the results of expertise. In attention strategy, I have developed the Eye Movement analysis with Hidden Markov Models method for quantifying eye movement pattern and consistency using both spatial and temporal information, which has led to novel findings across disciplines not discoverable using traditional methods. By integrating it with deep neural networks (DNN), I have developed DNN+HMM to account for eye movement strategy learning in human visual cognition. The understanding of the human mind through computational modeling also facilitates research on artificial intelligence's (AI) comparability with human cognition, which can in turn help explainable AI systems infer humans' belief on AI's operations and provide human-centered explanations to enhance human-AI interaction and mutual understanding. Together, these demonstrate the essential role of computational modeling methods in providing theoretical accounts of the human mind as well as its interaction with its environment and AI systems.

A sleepless night disrupts the resolution of emotional conflicts: Behavioural and neural evidence.

The present study aims to investigate the influence of 24-hr sleep deprivation on implicit emotion regulation using the emotional conflict task. Twenty-five healthy young adults completed a repeated-measures study protocol involving a night of at-home normal sleep control and a night of in-laboratory sleep deprivation. Prior to the experimental session, all participants wore an actigraph watch and completed the sleep diary. Following each condition, participants performed an emotional conflict task with electroencephalographic recordings. Emotional faces (fearful or happy) overlaid with words ("fear" or "happy") were used as stimuli creating congruent or incongruent trials, and participants were instructed to indicate whether the facial expression was happy or fearful. We measured the accuracy and reaction time on the emotional conflict task, as well as the mean amplitude of the P300 component of the event-related potential at CPz. At the behavioural level, sleep-deprived participants showed reduced alertness with overall longer reaction times and higher error rates. In addition, participants in the sleep deprivation condition made more errors when the current trial followed congruent trials compared with when it followed incongruent trials. At the neural level, P300 amplitude evoked under the sleep-deprived condition was significantly more positive compared with the normal sleep condition, and this effect interacted with previous-trial and current-trial congruency conditions, suggesting that participants used more attentional resources to resolve emotional conflicts when sleep deprived. Our study provided pioneering data demonstrating that sleep deprivation may impair the regulation of emotional processing in the absence of explicit instruction among emerging adults.

Music reading experience modulates eye movement pattern in English reading but not in Chinese reading.

Here we tested the hypothesis that in Chinese-English bilinguals, music reading experience may modulate eye movement planning in reading English but not Chinese sentences due to the similarity in perceptual demands on processing sequential symbol strings separated by spaces between music notation and English sentence reading. Chinese-English bilingual musicians and non-musicians read legal, semantically incorrect, and syntactically (and semantically) incorrect sentences in both English and Chinese. In English reading, musicians showed more dispersed eye movement patterns in reading syntactically incorrect sentences than legal sentences, whereas non-musicians did not. This effect was not observed in Chinese reading. Musicians also had shorter saccade lengths when viewing syntactically incorrect than correct musical notations and sentences in an unfamiliar alphabetic language (Tibetan), whereas non-musicians did not. Thus, musicians' eye movement planning was disturbed by syntactic violations in both music and English reading but not in Chinese reading, and this effect was generalized to an unfamiliar alphabetic language. These results suggested that music reading experience may modulate perceptual processes in reading differentially in bilinguals' two languages, depending on their processing similarities.

Non-monotonic developmental trend of holistic processing in visual expertise: the case of Chinese character recognition.

Holistic processing has been identified as an expertise marker of face and object recognition. By contrast, reduced holistic processing is purportedly an expertise marker in recognising orthographic characters in Chinese. Does holistic processing increase or decrease in expertise development? Is orthographic recognition a domain-specific exception to all other kinds of recognition (e.g. face and objects)? In two studies, we examined the developmental trend of holistic processing in Chinese character recognition in Chinese and non-Chinese children, and its relationship with literacy abilities: Chinese first graders-with emergent Chinese literacy acquired in kindergarten-showed increased holistic processing perhaps as an inchoate expertise marker when compared with kindergartners and non-Chinese first graders; however, the holistic processing effect was reduced in higher-grade Chinese children. These results suggest a non-monotonic inverted U-shape trend of holistic processing in visual expertise development: An increase in holistic processing due to initial reading experience followed by a decrease in holistic processing due to literacy enhancement. This result marks the development of holistic and analytic processing skills, both of which can be essential for mastering visual recognition. This study is the first to investigate the developmental trend of holistic processing in Chinese character recognition using the composite paradigm.

Impact of mask use on face recognition: an eye-tracking study.

We examined how mask use affects performance and eye movements in face recognition and whether strategy change reflected in eye movements is associated with performance change. Eighty-eight participants performed face recognition with masked faces either during learning only, during recognition only, or during both learning and recognition. As compared with the baseline condition where faces were unmasked during both learning and recognition, participants had impaired performance in all three scenarios, with larger impairment when mask conditions during learning and recognition did not match. When recognizing unmasked faces, whether the faces were learned with or without a mask on did not change eye movement behavior. Nevertheless, when recognizing unmasked faces that were learned with a mask on, participants who adopted more eyes-focused patterns had less performance impairment as compared with the baseline condition. When recognizing masked faces, participants had more eyes-focused patterns and more consistent gaze transition behavior than recognizing unmasked faces regardless of whether the faces were learned with or without a mask on. Nevertheless, when recognizing masked faces that were learned without a mask, participants whose gaze transition behavior was more consistent had less performance impairment as compared with the baseline condition. Thus, although eye movements during recognition were mainly driven by the mask condition during recognition but not that during learning, those who adjusted their strategy according to the mask condition difference between learning and recognition had better performance. This finding has important implications for identifying populations vulnerable to the impact of mask use and potential remedial strategies.

Holistic but with reduced right-hemisphere involvement: The case of dyslexia in Chinese character recognition.

Recent research on visual object recognition has suggested that the right hemisphere can engage either holistic or part-based processing depending on whether the recognition relies on configural (exact distances among features) or featural information, respectively. Consistent with this finding, expert Chinese reading has been marked by a left-side bias (an indication of right-hemisphere lateralization) with decreased holistic processing (as assessed using the composite paradigm) due to its reliance on featural information. Here we examine two common perceptual expertise phenomena in object recognition - holistic processing and left-side bias - of Chinese characters in adolescents with developmental dyslexia and matched controls. We found that those with dyslexia showed stronger holistic processing, a weaker left-side bias, and worse performance in Chinese character dictation than controls. This was in contrast to Limited writers (proficient readers with limited writing experience) reported in Tso, Au, and Hsiao (Psychological Science, 25, 1757-1767, 2014), who showed stronger holistic processing and worse dictation performance, but the same level of left-side bias as controls. This result demonstrated two different perceptual mechanisms underlying holistic processing: Limited writers' holistic processing may be due to difficulties in de-emphasizing configural information unimportant to Chinese characters, whereas readers with dyslexia may have deficits selectively attending to character components to form appropriate part-based representations in the right hemisphere.

Using emotion regulation strategies after sleep deprivation: ERP and behavioral findings.

Sleep deprivation is suggested to impact emotion regulation, but few studies have directly examined it. This study investigated the influence of sleep deprivation on three commonly used emotion regulation strategies (distraction, reappraisal, suppression) in Gross's (1998) process model of emotion regulation. Young healthy adults were randomly assigned to a sleep deprivation group (SD; n = 26, 13 males, age = 20.0 ± 1.7) or a sleep control group (SC; n = 25, 13 males, age = 20.2 ± 1.7). Following 24-h sleep deprivation or normal nighttime sleep, participants completed an emotion regulation task, in which they naturally viewed or applied a given emotion regulation strategy towards negative pictures, with electroencephalographic (EEG) recordings. A reduction in the centroparietal late positive potential (LPP) amplitudes towards negative pictures from the naturally viewing condition to a regulated condition was calculated as an index of regulatory effects. Comparisons between the two groups indicated that sleep deprivation significantly impaired the regulatory effects of distraction and reappraisal on LPP amplitudes. Suppression did not reduce LPP amplitudes in either group. In addition, habitual sleep quality moderated the effect of sleep deprivation on subjective perception of emotional stimuli, such that sleep deprivation only made good sleepers perceive negative pictures as more unpleasant and more arousing, but it had no significant effect on poor sleepers' perception of negative pictures. Altogether, this study provides the first evidence that sleep deprivation may impair the effectiveness of applying adaptive emotion regulation strategies (distraction and reappraisal), creating potentially undesirable consequences to emotional well-being.

Music reading expertise modulates hemispheric lateralization in English word processing but not in Chinese character processing.

Music notation and English word reading both involve mapping horizontally arranged visual components to components in sound, in contrast to reading in logographic languages such as Chinese. Accordingly, music-reading expertise may influence English word processing more than Chinese character processing. Here we showed that musicians named English words significantly faster than non-musicians when words were presented in the left visual field/right hemisphere (RH) or the center position, suggesting an advantage of RH processing due to music reading experience. This effect was not observed in Chinese character naming. A follow-up ERP study showed that in a sequential matching task, musicians had reduced RH N170 responses to English non-words under the processing of musical segments as compared with non-musicians, suggesting a shared visual processing mechanism in the RH between music notation and English non-word reading. This shared mechanism may be related to the letter-by-letter, serial visual processing that characterizes RH English word recognition (e.g., Lavidor & Ellis, 2001), which may consequently facilitate English word processing in the RH in musicians. Thus, music reading experience may have differential influences on the processing of different languages, depending on their similarities in the cognitive processes involved.

Perceptual expertise: can sensorimotor experience change holistic processing and left-side bias?

Holistic processing and left-side bias are both behavioral markers of expert face recognition. By contrast, expert recognition of characters in Chinese orthography involves left-side bias but reduced holistic processing, although faces and Chinese characters share many visual properties. Here, we examined whether this reduction in holistic processing of Chinese characters can be better explained by writing experience than by reading experience. Compared with Chinese nonreaders, Chinese readers who had limited writing experience showed increased holistic processing, whereas Chinese readers who could write characters fluently showed reduced holistic processing. This result suggests that writing and sensorimotor experience can modulate holistic-processing effects and that the reduced holistic processing observed in expert Chinese readers may depend mostly on writing experience. However, both expert writers and writers with limited experience showed similarly stronger left-side bias than novices did in processing mirror-symmetric Chinese characters; left-side bias may therefore be a robust expertise marker for object recognition that is uninfluenced by sensorimotor experience.

Visual field differences in visual word recognition can emerge purely from perceptual learning: evidence from modeling Chinese character pronunciation.

In Chinese orthography, a dominant character structure exists in which a semantic radical appears on the left and a phonetic radical on the right (SP characters); a minority opposite arrangement also exists (PS characters). As the number of phonetic radical types is much greater than semantic radical types, in SP characters the information is skewed to the right, whereas in PS characters it is skewed to the left. Through training a computational model for SP and PS character recognition that takes into account of the locations in which the characters appear in the visual field during learning, but does not assume any fundamental hemispheric processing difference, we show that visual field differences can emerge as a consequence of the fundamental structural differences in information between SP and PS characters, as opposed to the fundamental processing differences between the two hemispheres. This modeling result is also consistent with behavioral naming performance. This work provides strong evidence that perceptual learning, i.e., the information structure of word stimuli to which the readers have long been exposed, is one of the factors that accounts for hemispheric asymmetry effects in visual word recognition.

Two fixations suffice in face recognition.

It is well known that there exist preferred landing positions for eye fixations in visual word recognition. However, the existence of preferred landing positions in face recognition is less well established. It is also unknown how many fixations are required to recognize a face. To investigate these questions, we recorded eye movements during face recognition. During an otherwise standard face-recognition task, subjects were allowed a variable number of fixations before the stimulus was masked. We found that optimal recognition performance is achieved with two fixations; performance does not improve with additional fixations. The distribution of the first fixation is just to the left of the center of the nose, and that of the second fixation is around the center of the nose. Thus, these appear to be the preferred landing positions for face recognition. Furthermore, the fixations made during face learning differ in location from those made during face recognition and are also more variable in duration; this suggests that different strategies are used for face learning and face recognition.

Convergence of the visual field split: hemispheric modeling of face and object recognition.

Anatomical evidence shows that our visual field is initially split along the vertical midline and contralaterally projected to different hemispheres. It remains unclear at which processing stage the split information converges. In the current study, we applied the Double Filtering by Frequency (DFF) theory (Ivry & Robertson, 1998) to modeling the visual field split; the theory assumes a right-hemisphere/low-frequency bias. We compared three cognitive architectures with different timings of convergence and examined their cognitive plausibility to account for the left-side bias effect in face perception observed in human data. We show that the early convergence model failed to show the left-side bias effect. The modeling, hence, suggests that the convergence may take place at an intermediate or late stage, at least after information has been extracted/encoded separately in the two hemispheres, a fact that is often overlooked in computational modeling of cognitive processes. Comparative anatomical data suggest that this separate encoding process that results in differential frequency biases in the two hemispheres may be engaged from V1 up to the level of area V3a and V4v, and converge at least after the lateral occipital region. The left-side bias effect in our model was also observed in Greeble recognition; the modeling, hence, also provides testable predictions about whether the left-side bias effect may also be observed in (expertise-level) object recognition.

NIMBLE: a kernel density model of saccade-based visual memory.

We present a Bayesian version of J. Lacroix, J. Murre, and E. Postma's (2006) Natural Input Memory (NIM) model of saccadic visual memory. Our model, which we call NIMBLE (NIM with Bayesian Likelihood Estimation), uses a cognitively plausible image sampling technique that provides a foveated representation of image patches. We conceive of these memorized image fragments as samples from image class distributions and model the memory of these fragments using kernel density estimation. Using these models, we derive class-conditional probabilities of new image fragments and combine individual fragment probabilities to classify images. Our Bayesian formulation of the model extends easily to handle multi-class problems. We validate our model by demonstrating human levels of performance on a face recognition memory task and high accuracy on multi-category face and object identification. We also use NIMBLE to examine the change in beliefs as more fixations are taken from an image. Using fixation data collected from human subjects, we directly compare the performance of NIMBLE's memory component to human performance, demonstrating that using human fixation locations allows NIMBLE to recognize familiar faces with only a single fixation.

An examination of semantic radical combinability effects with lateralized cues in Chinese character recognition.

Auclair and Siéroff examined lateralized cuing effects in the identification of centrally presented letter strings and reported no cuing effects for short word stimuli. They argued for a redistribution of attention over the entire word for short familiar words. We explored cuing effects with Chinese phonetic compounds, which can be considered extreme examples of short words, in a character-level semantic judgment task. When the semantic radical position was placed on the left of the characters, strong radical combinability and semantic transparency effects were observed. There was also a significant interaction between cue position (left vs. right) and radical combinability: A left cue facilitated semantic judgment of characters with small radical combinability more than did a right cue. This behavior reflects the information profile of Chinese phonetic compounds. Semantic radicals with small combinability are more informative than those with large combinability in determining the meaning of the whole character; they therefore benefit more from a left than a right cue. A mechanism redistributing attention over the whole of the character was not in evidence at the level of semantic processing.

Neural correlates of foveal splitting in reading: evidence from an ERP study of Chinese character recognition.

Recent research on foveal structure and reading suggests that the two halves of a centrally fixated word seem to be initially projected to, and processed in, different hemispheres. In the current study, we utilize two contrasting structures in Chinese orthography, "SP" (the semantic radical on the left and the phonetic radical on the right) and "PS" characters (the opposite structure), to examine foveal splitting effects in event-related potential (ERP) recordings. We showed that when participants silently named centrally presented characters, there was a significant interaction between character type and hemisphere in N1 amplitude: SP characters elicited larger N1 compared with PS characters in the left hemisphere, whereas the right hemisphere had the opposite pattern. This effect is consistent with the split fovea claim, suggesting that the two halves of a character may be initially projected to and processed in different hemispheres. There was no such interaction observed in an earlier component P1. Also, there was an interaction between character type and sex of the reader in N350 amplitude. This result is consistent with Hsiao and Shillcock's [Hsiao, J. H., & Shillcock, R. (2005b). Foveal splitting causes differential processing of Chinese orthography in the male and female brain. Cognitive Brain Research, 25, 531-536] behavioural study, which showed a similar interaction in naming response time. They argued that this effect was due to a more left-lateralized network for phonological processing in the male brain compared with the female brain. The results hence showed that foveal splitting effects in visual word recognition were observed in N1 the earliest, and could extend far enough to interact with the sex of the reader as revealed in N350.

Analysis of a Chinese phonetic compound database: implications for orthographic processing.

The complexity of Chinese orthography has hindered the progress of research in Chinese to the same level of sophistication of that in alphabetic languages such as English. Also, there has been no publicly available resource concerning the decomposition of Chinese characters, which is essential in any attempt to model the cognitive processes of Chinese character recognition. Here we report our construction and analysis of a Chinese lexical database containing the most frequent phonetic compounds decomposed into semantic and phonetic radicals according to Chinese etymology. Each radical was further decomposed into basic stroke patterns according to a Chinese transcription system, Cangjie (Chu, 1979 Laboratory of chu Bong-Foo Retrieved August 25, 2004, from http://www.cbflabs.com/). Other information such as pronunciation and character frequency were also incorporated. We examine the distribution of different types of character, the information skew in phonetic compounds, the relations between subcharacter orthographic units and the pronunciation of the entire character, and the processing implications of these phenomena in terms of universal psycholinguistic principles.

A TMS examination of semantic radical combinability effects in Chinese character recognition.

The proposal of human foveal splitting assumes a vertical meridian split in the foveal representation and the consequent contralateral projection of information in the two hemifields to the two hemispheres and has been shown to have important implications for visual word recognition. According to this assumption, in Chinese character recognition, the two halves of a centrally fixated character may be initially projected to and processed in different hemispheres. Here, we describe a repetitive transcranial magnetic stimulation (rTMS) investigation of hemispheric processing in Chinese character recognition, through examining semantic radical combinability effects in a character semantic judgment task. The materials used were a dominant type of Chinese character which consists of a semantic radical on the left and a phonetic radical on the right. Thus, according to the split fovea assumption, the semantic and phonetic radicals are initially projected to and processed in the right hemisphere and the left hemisphere, respectively. We show that rTMS over the left occipital cortex impaired the facilitation of semantic radicals with large combinability, whereas right occipital rTMS did not. This interaction between stimulation site and radical combinability reveals a flexible division of labor between the hemispheres in Chinese character recognition, with each hemisphere responding optimally to the information in the contralateral visual hemifield to which it has direct access. The results are also consistent with the split fovea claim, suggesting functional foveal splitting as a universal processing constraint in reading.

Foveal splitting causes differential processing of Chinese orthography in the male and female brain.

Chinese characters contain separate phonetic and semantic radicals. A dominant character type exists in which the semantic radical is on the left and the phonetic radical on the right; an opposite, minority structure also exists, with the semantic radical on the right and the phonetic radical on the left. We show that, when asked to pronounce isolated tokens of these two character types, males responded significantly faster when the phonetic information was on the right, whereas females showed a non-significant tendency in the opposite direction. Recent research on foveal structure and reading suggests that the two halves of a centrally fixated character are initially processed in different hemispheres. The male brain typically relies more on the left hemisphere for phonological processing compared with the female brain, causing this gender difference to emerge. This interaction is predicted by an implemented computational model. This study supports the existence of a gender difference in phonological processing, and shows that the effects of foveal splitting in reading extend far enough into word recognition to interact with the gender of the reader in a naturalistic reading task.