Linguistic Empathy: Behavioral measures, neurophysiological correlates, and correlation with Psychological Empathy.

Relations among behavioral, psychological, and electrophysiological correlates of Linguistic Empathy were examined in two experiments using lateralized stimuli. Linguistic Empathy is defined as a linguistic manifestation of the point of view the speaker assumes toward the content of the utterance, and of the speaker's attitude toward/identification with the referents therein. Linguistic choices made by the speaker among multiple logically and referentially synonymous lexical and grammatical options reveal the speaker's perspectives. In experiment 1, acceptability ratings were measured for Context-Target sentence pairs that did or did not violate two Empathy Hierarchies (Person Empathy Hierarchy and Topic Empathy Hierarchy); the Empathy Quotient (EQ) test of Psychological Empathy was also administered. Ratings were lower for sentence pairs that violated both hierarchies than for those violating neither and were intermediate for sentences violating only one hierarchy. Linguistic Empathy (LE) was operationalized as the difference in ratings between sentences violating both vs. neither empathy hierarchy; this measure correlated positively with EQ. Experiment 2 replicated those results with new participants and measured reaction time and EEG during ratings. While there were no effects of hemisphere or visual field on the linguistic variables, the amplitude of a positive event-related potential deflection at 380 ms provided a partial electrophysiological correlate for LE. Its difference measure correlated with behavioral LE but not with EQ. Though preliminary, these experiments show that Linguistic Empathy may share information processing computations with Psychological Empathy and have an electrophysiological correlate.

Reduced direction discrimination sensitivity in visual motion adaptation, and the role of perceptual learning.

We investigated visual direction discrimination under the influence of motion aftereffect (MAE). Participants in each experiment first adapted to a horizontally drifting grating before deciding whether a drifting test grating moved to the left or right. A psychometric function was obtained as a function of the velocity of the test. Interestingly, in addition to the horizontal shift of the psychometric function that typified the MAE, the slope of the psychometric function became shallower after adaptation, indicating decreased discrimination sensitivity. However, this decrease was only observed in psychophysically experienced participants. Motivated, but psychophysically inexperienced participants only showed this effect after weeks of perceptual learning. This shallowing effect transferred to the untrained adaptation direction (e.g., from leftward adaptation to rightward), although perceptual learning of improved discrimination could not transfer. When the test duration was lengthened to reduce task difficulty, less training was needed to produce the same effect. These results indicate that, post-adaptation and when steady measurements could be obtained, left-right motion direction discrimination sensitivity was reduced.

Local features and global shape information in object classification by deep convolutional neural networks.

Deep convolutional neural networks (DCNNs) show impressive similarities to the human visual system. Recent research, however, suggests that DCNNs have limitations in recognizing objects by their shape. We tested the hypothesis that DCNNs are sensitive to an object's local contour features but have no access to global shape information that predominates human object recognition. We employed transfer learning to assess local and global shape processing in trained networks. In Experiment 1, we used restricted and unrestricted transfer learning to retrain AlexNet, VGG-19, and ResNet-50 to classify circles and squares. We then probed these networks with stimuli with conflicting global shape and local contour information. We presented networks with overall square shapes comprised of curved elements and circles comprised of corner elements. Networks classified the test stimuli by local contour features rather than global shapes. In Experiment 2, we changed the training data to include circles and squares comprised of different elements so that the local contour features of the object were uninformative. This considerably increased the network's tendency to produce global shape responses, but deeper analyses in Experiment 3 revealed the network still showed no sensitivity to the spatial configuration of local elements. These findings demonstrate that DCNNs' performance is an inversion of human performance with respect to global and local shape processing. Whereas abstract relations of elements predominate in human perception of shape, DCNNs appear to extract only local contour fragments, with no representation of how they spatially relate to each other to form global shapes.

Object-based attention generalizes to multisurface objects.

When a part of an object is cued, targets presented in other locations on the same object are detected more rapidly and accurately than are targets on other objects. Often in object-based attention experiments, cues and targets appear not only on the same object but also on the same surface. In four psychophysical experiments, we examined whether the "object" of attentional selection was the entire object or one of its surfaces. In Experiment 1, facilitation effects were found for targets on uncued, adjacent surfaces on the same object, even when the cued and uncued surfaces were oriented differently in depth. This suggests that the "object-based" benefits of attention are not restricted to individual surfaces. Experiments 2a and 2b examined the interaction of perceptual grouping and object-based attention. In both experiments, cuing benefits extended across objects when the surfaces of those objects could be grouped, but the effects were not as strong as in Experiment 1, where the surfaces belonged to the same object. The cuing effect was strengthened in Experiment 3 by connecting the cued and target surfaces with an intermediate surface, making them appear to all belong to the same object. Together, the experiments suggest that the objects of attention do not necessarily map onto discrete physical objects defined by bounded surfaces. Instead, attentional selection can be allocated to perceptual groups of surfaces and objects in the same way as it can to a location or to groups of features that define a single object.

The Wandering Circles: A Flicker Rate and Contour-Dependent Motion Illusion.

Understanding of the visual system can be informed by examining errors in perception. We present a novel illusion-Wandering Circles-in which stationary circles undergoing contrast-polarity reversals (i.e., flicker), when viewed peripherally, appear to move about in a random fashion. In two psychophysical experiments, participants rated the strength of perceived illusory motion under varying stimulus conditions. The illusory motion percept was strongest when the circle's edge was defined by a light/dark alternation and when the edge faded smoothly to the background gray (i.e., a circular arrangement of the Craik-O'Brien-Cornsweet illusion). In addition, the percept of illusory motion is flicker rate dependent, appearing strongest when the circles reversed polarity 9.44 times per second and weakest at 1.98 times per second. The Wandering Circles differ from many other classic motion illusions as the light/dark alternation is perfectly balanced in time and position around the edges of the circle, and thus, there is no net directional local or global motion energy in the stimulus. The perceived motion may instead rely on factors internal to the viewer such as top-down influences, asymmetries in luminance and motion perception across the retina, adaptation combined with positional uncertainty due to peripheral viewing, eye movements, or low contrast edges.

The effect of perceptual contour orientation uncertainty on the tilt aftereffect.

The tilt aftereffect (TAE) occurs when, after adapting to an oriented line, a vertical line appears to be tilted in the opposite direction. The magnitude of the TAE has been shown to relate to the salience of the adapting stimulus (e.g., its contrast) as well as to the similarity between the adapting and testing stimuli. However, the relationship between TAE and orientation uncertainty - variability in the perceived orientation of the stimulus - of either the adapting or the testing stimulus and, more importantly, change in orientation uncertainty as a function of adaptation have not previously been explored. We manipulated stimulus salience by using a variety of contour types, including real and illusory contours. Tilt aftereffects were observed even for stimuli that had much weaker or invisible illusory contours. Orientation uncertainty of the adapting stimulus, as measured by the slope of a psychometric function in orientation discrimination, was positively correlated with TAE magnitude for real and illusory contours, but not for stimuli with weak contour percepts. On an individual subject level, orientation uncertainty increased post-adaptation and was correlated with pre-adaptation uncertainty. That is, individuals with more variability in their perception of orientation before adaptation showed increased variability in orientation discrimination following adaptation. This may account for some of the variability in TAE across individuals and stimulus types and is consistent with previous findings on increased orientation discrimination thresholds post-adaptation for nearby orientations.

The motion-induced contour revisited: Observations on 3-D structure and illusory contour formation in moving stimuli.

The motion-induced contour (MIC) was first described by Victor Klymenko and Naomi Weisstein in a series of papers in the 1980s. The effect is created by rotating the outline of a tilted cube in depth. When one of the vertical edges is removed, an illusory contour can be seen in its place. In four experiments, we explored which stimulus features influence perceived illusory contour strength. Participants provided subjective ratings of illusory contour strength as a function of orientation of the stimulus, separation between inducing edges, and the length of inducing edges. We found that the angle of tilt of the object in depth had the largest impact on perceived illusory contour strength with tilt angles of 20° and 30° producing the strongest percepts. Tilt angle is an unexplored feature of structure-from-motion displays. In addition, we found that once the depth structure of the object was extracted, other features of the display, such as the distance spanned by the illusory contour, could also influence its strength, similar to the notion of support ratio for 2-D illusory contours. Illusory contour strength was better predicted by the length of the contour in 3-D rather than in 2-D, suggesting that MICs are constructed by a 3-D process that takes as input initially recovered contour orientation and position information in depth and only then forms interpolations between them.

Interactions of flicker and motion.

We present a series of novel observations about interactions between flicker and motion that lead to three distinct perceptual effects. We use the term flicker to describe alternating changes in a stimulus' luminance or color (i.e. a circle that flickers from black to white and visa-versa). When objects flicker, three distinct phenomena can be observed: (1) Flicker Induced Motion (FLIM) in which a single, stationary object, appears to move when it flickers at certain rates; (2) Flicker Induced Motion Suppression (FLIMS) in which a moving object appears to be stationary when it flickers at certain rates, and (3) Flicker-Induced Induced-Motion (FLIIM) in which moving objects that are flickering induce another flickering stationary object to appear to move. Across four psychophysical experiments, we characterize key stimulus parameters underlying these flicker-motion interactions. Interactions were strongest in the periphery and at flicker frequencies above 10 Hz. Induced motion occurred not just for luminance flicker, but for isoluminant color changes as well. We also found that the more physically moving objects there were, the more motion induction to stationary objects occurred. We present demonstrations that the effects reported here cannot be fully accounted for by eye movements: we show that the perceived motion of multiple stationary objects that are induced to move via flicker can appear to move independently and in random directions, whereas eye movements would have caused all of the objects to appear to move coherently. These effects highlight the fundamental role of spatiotemporal dynamics in the representation of motion and the intimate relationship between flicker and motion.

Deep convolutional networks do not classify based on global object shape.

Deep convolutional networks (DCNNs) are achieving previously unseen performance in object classification, raising questions about whether DCNNs operate similarly to human vision. In biological vision, shape is arguably the most important cue for recognition. We tested the role of shape information in DCNNs trained to recognize objects. In Experiment 1, we presented a trained DCNN with object silhouettes that preserved overall shape but were filled with surface texture taken from other objects. Shape cues appeared to play some role in the classification of artifacts, but little or none for animals. In Experiments 2-4, DCNNs showed no ability to classify glass figurines or outlines but correctly classified some silhouettes. Aspects of these results led us to hypothesize that DCNNs do not distinguish object's bounding contours from other edges, and that DCNNs access some local shape features, but not global shape. In Experiment 5, we tested this hypothesis with displays that preserved local features but disrupted global shape, and vice versa. With disrupted global shape, which reduced human accuracy to 28%, DCNNs gave the same classification labels as with ordinary shapes. Conversely, local contour changes eliminated accurate DCNN classification but caused no difficulty for human observers. These results provide evidence that DCNNs have access to some local shape information in the form of local edge relations, but they have no access to global object shapes.

Towards a unified perspective of object shape and motion processing in human dorsal cortex.

Although object-related areas were discovered in human parietal cortex a decade ago, surprisingly little is known about the nature and purpose of these representations, and how they differ from those in the ventral processing stream. In this article, we review evidence for the unique contribution of object areas of dorsal cortex to three-dimensional (3-D) shape representation, the localization of objects in space, and in guiding reaching and grasping actions. We also highlight the role of dorsal cortex in form-motion interaction and spatiotemporal integration, possible functional relationships between 3-D shape and motion processing, and how these processes operate together in the service of supporting goal-directed actions with objects. Fundamental differences between the nature of object representations in the dorsal versus ventral processing streams are considered, with an emphasis on how and why dorsal cortex supports veridical (rather than invariant) representations of objects to guide goal-directed hand actions in dynamic visual environments.

The maintenance and updating of representations of no longer visible objects and their parts.

When an object partially or completely disappears behind an occluding surface, a representation of that object persists. For example, fragments of no longer visible objects can serve as an input into mid-level constructive visual processes, interacting and integrating with currently visible portions to form perceptual units and global motion signals. Remarkably, these persistent representations need not be static and can have their positions and orientations updated postdictively as new information becomes visible. In this chapter, we highlight historical considerations, behavioral evidence, and neural correlates of this type of representational updating of no longer visible information at three distinct levels of visual processing. At the lowest level, we discuss spatiotemporal boundary formation in which visual transients can be integrated over space and time to construct local illusory edges, global form, and global motion percepts. At an intermediate level, we review how the visual system updates form information seen at one moment in time and integrates it with subsequently available information to generate global shape and motion representations (e.g., spatiotemporal form integration and anorthoscopic perception). At a higher level, when an entire object completely disappears behind an occluder, the object's identity and predicted position can be maintained in the absence of visual information.

Decoding information about dynamically occluded objects in visual cortex.

During dynamic occlusion, an object passes behind an occluding surface and then later reappears. Even when completely occluded from view, such objects are experienced as continuing to exist or persist behind the occluder even though they are no longer visible. The contents and neural basis of this persistent representation remain poorly understood. Questions remain as to whether there is information maintained about the object itself (i.e. its shape or identity) or non-object-specific information such as its position or velocity as it is tracked behind an occluder, as well as which areas of visual cortex represent such information. Recent studies have found that early visual cortex is activated by "invisible" objects during visual imagery and by unstimulated regions along the path of apparent motion, suggesting that some properties of dynamically occluded objects may also be neurally represented in early visual cortex. We applied functional magnetic resonance imaging in human subjects to examine representations within visual cortex during dynamic occlusion. For gradually occluded, but not for instantly disappearing objects, there was an increase in activity in early visual cortex (V1, V2, and V3). This activity was spatially-specific, corresponding to the occluded location in the visual field. However, the activity did not encode enough information about object identity to discriminate between different kinds of occluded objects (circles vs. stars) using MVPA. In contrast, object identity could be decoded in spatially-specific subregions of higher-order, topographically organized areas such as ventral, lateral, and temporal occipital areas (VO, LO, and TO) as well as the functionally defined LOC and hMT+. These results suggest that early visual cortex may only represent the dynamically occluded object's position or motion path, while later visual areas represent object-specific information.

From Flashes to Edges to Objects: Recovery of Local Edge Fragments Initiates Spatiotemporal Boundary Formation.

Spatiotemporal boundary formation (SBF) is the perception of illusory boundaries, global form, and global motion from spatially and temporally sparse transformations of texture elements (Shipley and Kellman, 1993a, 1994; Erlikhman and Kellman, 2015). It has been theorized that the visual system uses positions and times of element transformations to extract local oriented edge fragments, which then connect by known interpolation processes to produce larger contours and shapes in SBF. To test this theory, we created a novel display consisting of a sawtooth arrangement of elements that disappeared and reappeared sequentially. Although apparent motion along the sawtooth would be expected, with appropriate spacing and timing, the resulting percept was of a larger, moving, illusory bar. This display approximates the minimal conditions for visual perception of an oriented edge fragment from spatiotemporal information and confirms that such events may be initiating conditions in SBF. Using converging objective and subjective methods, experiments showed that edge formation in these displays was subject to a temporal integration constraint of ~80 ms between element disappearances. The experiments provide clear support for models of SBF that begin with extraction of local edge fragments, and they identify minimal conditions required for this process. We conjecture that these results reveal a link between spatiotemporal object perception and basic visual filtering. Motion energy filters have usually been studied with orientation given spatially by luminance contrast. When orientation is not given in static frames, these same motion energy filters serve as spatiotemporal edge filters, yielding local orientation from discrete element transformations over time. As numerous filters of different characteristic orientations and scales may respond to any simple SBF stimulus, we discuss the aperture and ambiguity problems that accompany this conjecture and how they might be resolved by the visual system.

The neural representation of objects formed through the spatiotemporal integration of visual transients.

Oftentimes, objects are only partially and transiently visible as parts of them become occluded during observer or object motion. The visual system can integrate such object fragments across space and time into perceptual wholes or spatiotemporal objects. This integrative and dynamic process may involve both ventral and dorsal visual processing pathways, along which shape and spatial representations are thought to arise. We measured fMRI BOLD response to spatiotemporal objects and used multi-voxel pattern analysis (MVPA) to decode shape information across 20 topographic regions of visual cortex. Object identity could be decoded throughout visual cortex, including intermediate (V3A, V3B, hV4, LO1-2,) and dorsal (TO1-2, and IPS0-1) visual areas. Shape-specific information, therefore, may not be limited to early and ventral visual areas, particularly when it is dynamic and must be integrated. Contrary to the classic view that the representation of objects is the purview of the ventral stream, intermediate and dorsal areas may play a distinct and critical role in the construction of object representations across space and time.

Modeling spatiotemporal boundary formation.

Spatiotemporal boundary formation (SBF) refers to perception of continuous contours, shape, and global motion from sequential transformations of widely separated surface elements. How such minimal information in SBF can produce whole forms and the nature of the computational processes involved remain mysterious. Formally, it has been shown that orientations and motion directions of local edge fragments can be recovered from small sets of element changes (Shipley & Kellman, (1997). Vision Research, 37, 1281-1293). Little experimental work has examined SBF in simple situations, however, and no model has been able to predict human SBF performance. We measured orientation discrimination thresholds in simple SBF displays for thin, oriented bars as a function of element density, number of element transformations, and frame duration. Thresholds decreased with increasing density and number of transformations, and increased with frame duration. An ideal observer model implemented to give trial-by-trial responses in the same orientation discrimination task exceeded human performance. In a second group of experiments, we measured human precision in detecting inputs to the model (spatial, temporal, and angular inter-element separation). A model that modified the ideal observer by added encoding imprecision for these parameters, directly obtained from Exp. 2, and that included two integration constraints obtained from previous research, closely fit human SBF data with no additional free parameters. These results provide the first empirical support for an early stage in shape formation in SBF based on the recovery of local edge fragments from spatiotemporally sparse element transformation events.

Multiple forms of contour grouping deficits in schizophrenia: what is the role of spatial frequency?

Schizophrenia patients poorly perceive Kanizsa figures and integrate co-aligned contour elements (Gabors). They also poorly process low spatial frequencies (SFs), which presumably reflects dysfunction along the dorsal pathway. Can contour grouping deficits be explained in terms of the spatial frequency content of the display elements? To address the question, we tested patients and matched controls on three contour grouping paradigms in which the SF composition was modulated. In the Kanizsa task, subjects discriminated quartets of sectored circles ("pac-men") that either formed or did not form Kanizsa shapes (illusory and fragmented conditions, respectively). In contour integration, subjects identified the screen quadrant thought to contain a closed chain of co-circular Gabors. In collinear facilitation, subjects attempted to detect a central low-contrast element flanked by collinear or orthogonal high-contrast elements, and facilitation corresponded to the amount by which collinear flankers reduced contrast thresholds. We varied SF by modifying the element features in the Kanizsa task and by scaling the entire stimulus display in the remaining tasks (SFs ranging from 4 to 12 cycles/deg). Irrespective of SF, patients were worse at discriminating illusory, but not fragmented shapes. Contrary to our hypothesis, collinear facilitation and contour integration were abnormal in the clinical group only for the higher SF (>=10 c/deg). Grouping performance correlated with clinical variables, such as conceptual disorganization, general symptoms, and levels of functioning. In schizophrenia, three forms of contour grouping impairments prominently arise and cannot be attributed to poor low SF processing. Neurobiological and clinical implications are discussed.

Forensic comparison and matching of fingerprints: using quantitative image measures for estimating error rates through understanding and predicting difficulty.

Latent fingerprint examination is a complex task that, despite advances in image processing, still fundamentally depends on the visual judgments of highly trained human examiners. Fingerprints collected from crime scenes typically contain less information than fingerprints collected under controlled conditions. Specifically, they are often noisy and distorted and may contain only a portion of the total fingerprint area. Expertise in fingerprint comparison, like other forms of perceptual expertise, such as face recognition or aircraft identification, depends on perceptual learning processes that lead to the discovery of features and relations that matter in comparing prints. Relatively little is known about the perceptual processes involved in making comparisons, and even less is known about what characteristics of fingerprint pairs make particular comparisons easy or difficult. We measured expert examiner performance and judgments of difficulty and confidence on a new fingerprint database. We developed a number of quantitative measures of image characteristics and used multiple regression techniques to discover objective predictors of error as well as perceived difficulty and confidence. A number of useful predictors emerged, and these included variables related to image quality metrics, such as intensity and contrast information, as well as measures of information quantity, such as the total fingerprint area. Also included were configural features that fingerprint experts have noted, such as the presence and clarity of global features and fingerprint ridges. Within the constraints of the overall low error rates of experts, a regression model incorporating the derived predictors demonstrated reasonable success in predicting objective difficulty for print pairs, as shown both in goodness of fit measures to the original data set and in a cross validation test. The results indicate the plausibility of using objective image metrics to predict expert performance and subjective assessment of difficulty in fingerprint comparisons.

Non-rigid illusory contours and global shape transformations defined by spatiotemporal boundary formation.

Spatiotemporal boundary formation (SBF) is the perception of form, global motion, and continuous boundaries from relations of discrete changes in local texture elements (Shipley and Kellman, 1994). In two experiments, small, circular elements underwent small displacements whenever an edge of an invisible (virtual) object passed over them. Unlike previous studies that examined only rigidly translating objects, we tested virtual objects whose properties changed continuously. Experiment 1 tested rigid objects that changed in orientation, scale, and velocity. Experiment 2 tested objects that transformed non-rigidly taking on a series of shapes. Robust SBF occurred for all of the rigid transformations tested, as well as for non-rigid virtual objects, producing the perception of continuously bounded, smoothly deforming shapes. These novel illusions involve perhaps the most extreme cases of visual perception of continuous boundaries and shape from minimal information. They show that SBF encompasses a wider range of illusory phenomena than previously understood, and they present substantial challenges for existing models of SBF.

Automatic feature-based grouping during multiple object tracking.

Contour interpolation automatically binds targets with distractors to impair multiple object tracking (Keane, Mettler, Tsoi, & Kellman, 2011). Is interpolation special in this regard or can other features produce the same effect? To address this question, we examined the influence of eight features on tracking: color, contrast polarity, orientation, size, shape, depth, interpolation, and a combination (shape, color, size). In each case, subjects tracked 4 of 8 objects that began as undifferentiated shapes, changed features as motion began (to enable grouping), and returned to their undifferentiated states before halting. We found that intertarget grouping improved performance for all feature types except orientation and interpolation (Experiment 1 and Experiment 2). Most importantly, target-distractor grouping impaired performance for color, size, shape, combination, and interpolation. The impairments were, at times, large (>15% decrement in accuracy) and occurred relative to a homogeneous condition in which all objects had the same features at each moment of a trial (Experiment 2), and relative to a "diversity" condition in which targets and distractors had different features at each moment (Experiment 3). We conclude that feature-based grouping occurs for a variety of features besides interpolation, even when irrelevant to task instructions and contrary to the task demands, suggesting that interpolation is not unique in promoting automatic grouping in tracking tasks. Our results also imply that various kinds of features are encoded automatically and in parallel during tracking.