Subtle adversarial image manipulations influence both human and machine perception.

Although artificial neural networks (ANNs) were inspired by the brain, ANNs exhibit a brittleness not generally observed in human perception. One shortcoming of ANNs is their susceptibility to adversarial perturbations-subtle modulations of natural images that result in changes to classification decisions, such as confidently mislabelling an image of an elephant, initially classified correctly, as a clock. In contrast, a human observer might well dismiss the perturbations as an innocuous imaging artifact. This phenomenon may point to a fundamental difference between human and machine perception, but it drives one to ask whether human sensitivity to adversarial perturbations might be revealed with appropriate behavioral measures. Here, we find that adversarial perturbations that fool ANNs similarly bias human choice. We further show that the effect is more likely driven by higher-order statistics of natural images to which both humans and ANNs are sensitive, rather than by the detailed architecture of the ANN.

Predicting the Ease of Human Category Learning Using Radial Basis Function Networks.

Our goal is to understand and optimize human concept learning by predicting the ease of learning of a particular exemplar or category. We propose a method for estimating ease values, quantitative measures of ease of learning, as an alternative to conducting costly empirical training studies. Our method combines a psychological embedding of domain exemplars with a pragmatic categorization model. The two components are integrated using a radial basis function network (RBFN) that predicts ease values. The free parameters of the RBFN are fit using human similarity judgments, circumventing the need to collect human training data to fit more complex models of human categorization. We conduct two category-training experiments to validate predictions of the RBFN. We demonstrate that an instance-based RBFN outperforms both a prototype-based RBFN and an empirical approach using the raw data. Although the human data were collected across diverse experimental conditions, the predicted ease values strongly correlate with human learning performance. Training can be sequenced by (predicted) ease, achieving what is known as fading in the psychology literature and curriculum learning in the machine-learning literature, both of which have been shown to facilitate learning.

Obtaining psychological embeddings through joint kernel and metric learning.

Psychological embeddings provide a powerful formalism for characterizing human-perceived similarity among members of a stimulus set. Obtaining high-quality embeddings can be costly due to algorithm design, software deployment, and participant compensation. This work aims to advance state-of-the-art embedding techniques and provide a comprehensive software package that makes obtaining high-quality psychological embeddings both easy and relatively efficient. Contributions are made on four fronts. First, the embedding procedure allows multiple trial configurations (e.g., triplets) to be used for collecting similarity judgments from participants. For example, trials can be configured to collect triplet comparisons or to sort items into groups. Second, a likelihood model is provided for three classes of similarity kernels allowing users to easily infer the parameters of their preferred model using gradient descent. Third, an active selection algorithm is provided that makes data collection more efficient by proposing comparisons that provide the strongest constraints on the embedding. Fourth, the likelihood model allows the specification of group-specific attention weight parameters. A series of experiments are included to highlight each of these contributions and their impact on converging to a high-quality embedding. Collectively, these incremental improvements provide a powerful and complete set of tools for inferring psychological embeddings. The relevant tools are available as the Python package PsiZ, which can be cloned from GitHub ( https://github.com/roads/psiz ).

Rejecting salient distractors: Generalization from experience.

Distraction impairs performance of many important, everyday tasks. Attentional control limits distraction by preferentially selecting important items for limited-capacity cognitive operations. Research in attentional control has typically investigated the degree to which selection of items is stimulus-driven versus goal-driven. Recent work finds that when observers initially learn a task, the selection is based on stimulus-driven factors, but through experience, goal-driven factors have an increasing influence. The modulation of selection by goals has been studied within the paradigm of learned distractor rejection, in which experience over a sequence of trials enables individuals eventually to ignore a perceptually salient distractor. The experiments presented examine whether observers can generalize learned distractor rejection to novel distractors. Observers searched for a target and ignored a salient color-singleton distractor that appeared in half of the trials. In Experiment 1, observers who learned distractor rejection in a variable environment rejected a novel distractor more effectively than observers who learned distractor rejection in a less variable, homogeneous environment, demonstrating that variable, heterogeneous stimulus environments encourage generalizable learned distractor rejection. Experiments 2 and 3 investigated the time course of learned distractor rejection across the experiment and found that after experiencing four color-singleton distractors in different blocks, observers could effectively reject subsequent novel color-singleton distractors. These results suggest that the optimization of attentional control to the task environment can be interpreted as a form of learning, demonstrating experience's critical role in attentional control.

Forgetting of Foreign-Language Skills: A Corpus-Based Analysis of Online Tutoring Software.

We explore the nature of forgetting in a corpus of 125,000 students learning Spanish using the Rosetta Stone® foreign-language instruction software across 48 lessons. Students are tested on a lesson after its initial study and are then retested after a variable time lag. We observe forgetting consistent with power function decay at a rate that varies across lessons but not across students. We find that lessons which are better learned initially are forgotten more slowly, a correlation which likely reflects a latent cause such as the quality or difficulty of the lesson. We obtain improved predictive accuracy of the forgetting model by augmenting it with features that encode characteristics of a student's initial study of the lesson and the activities the student engaged in between the initial and delayed tests. The augmented model can predict 23.9% of the variance in an individual's score on the delayed test. We analyze which features best explain individual performance.

Improving Human-Machine Cooperative Classification Via Cognitive Theories of Similarity.

Acquiring perceptual expertise is slow and effortful. However, untrained novices can accurately make difficult classification decisions (e.g., skin-lesion diagnosis) by reformulating the task as similarity judgment. Given a query image and a set of reference images, individuals are asked to select the best matching reference. When references are suitably chosen, the procedure yields an implicit classification of the query image. To optimize reference selection, we develop and evaluate a predictive model of similarity-based choice. The model builds on existing psychological literature and accommodates stochastic, dynamic shifts of attention among visual feature dimensions. We perform a series of human experiments with two stimulus types (rectangles, faces) and nine classification tasks to validate the model and to demonstrate the model's potential to boost performance. Our system achieves high accuracy for participants who are naive as to the classification task, even when the classification task switches from trial to trial.

Using Highlighting to Train Attentional Expertise.

Acquiring expertise in complex visual tasks is time consuming. To facilitate the efficient training of novices on where to look in these tasks, we propose an attentional highlighting paradigm. Highlighting involves dynamically modulating the saliency of a visual image to guide attention along the fixation path of a domain expert who had previously viewed the same image. In Experiment 1, we trained naive subjects via attentional highlighting on a fingerprint-matching task. Before and after training, we asked subjects to freely inspect images containing pairs of prints and determine whether the prints matched. Fixation sequences were automatically scored for the degree of expertise exhibited using a Bayesian discriminative model of novice and expert gaze behavior. Highlighted training causes gaze behavior to become more expert-like not only on the trained images but also on transfer images, indicating generalization of learning. In Experiment 2, to control for the possibility that the increase in expertise is due to mere exposure, we trained subjects via highlighting of fixation sequences from novices, not experts, and observed no transition toward expertise. In Experiment 3, to determine the specificity of the training effect, we trained subjects with expert fixation sequences from images other than the one being viewed, which preserves coarse-scale statistics of expert gaze but provides no information about fine-grain features. Observing at least a partial transition toward expertise, we obtain only weak evidence that the highlighting procedure facilitates the learning of critical local features. We discuss possible improvements to the highlighting procedure.

Retrieval practice over the long term: should spacing be expanding or equal-interval?

If multiple opportunities are available to review to-be-learned material, should a review occur soon after initial study and recur at progressively expanding intervals, or should the reviews occur at equal intervals? Landauer and Bjork (1978) argued for the superiority of expanding intervals, whereas more recent research has often failed to find any advantage. However, these prior studies have generally compared expanding versus equal-interval training within a single session, and have assessed effects only upon a single final test. We argue that a more generally important goal would be to maintain high average performance over a considerable period of training. For the learning of foreign vocabulary spread over four weeks, we found that expanding retrieval practice (i.e., sessions separated by increasing numbers of days) produced recall equivalent to that from equal-interval practice on a final test given eight weeks after training. However, the expanding schedule yielded much higher average recallability over the whole training period.

Maximizing students' retention via spaced review: practical guidance from computational models of memory.

During each school semester, students face an onslaught of material to be learned. Students work hard to achieve initial mastery of the material, but when they move on, the newly learned facts, concepts, and skills degrade in memory. Although both students and educators appreciate that review can help stabilize learning, time constraints result in a trade-off between acquiring new knowledge and preserving old knowledge. To use time efficiently, when should review take place? Experimental studies have shown benefits to long-term retention with spaced study, but little practical advice is available to students and educators about the optimal spacing of study. The dearth of advice is due to the challenge of conducting experimental studies of learning in educational settings, especially where material is introduced in blocks over the time frame of a semester. In this study, we turn to two established models of memory-ACT-R and MCM-to conduct simulation studies exploring the impact of study schedule on long-term retention. Based on the premise of a fixed time each week to review, converging evidence from the two models suggests that an optimal review schedule obtains significant benefits over haphazard (suboptimal) review schedules. Furthermore, we identify two scheduling heuristics that obtain near optimal review performance: (a) review the material from µ-weeks back, and (b) review material whose predicted memory strength is closest to a particular threshold. The former has implications for classroom instruction and the latter for the design of digital tutors.

Improving students' long-term knowledge retention through personalized review.

Human memory is imperfect; thus, periodic review is required for the long-term preservation of knowledge and skills. However, students at every educational level are challenged by an ever-growing amount of material to review and an ongoing imperative to master new material. We developed a method for efficient, systematic, personalized review that combines statistical techniques for inferring individual differences with a psychological theory of memory. The method was integrated into a semester-long middle-school foreign-language course via retrieval-practice software. Using a cumulative exam administered after the semester's end, we compared time-matched review strategies and found that personalized review yielded a 16.5% boost in course retention over current educational practice (massed study) and a 10.0% improvement over a one-size-fits-all strategy for spaced study.

Sequential effects in response time reveal learning mechanisms and event representations.

Binary choice tasks, such as 2-alternative forced choice, show a complex yet consistent pattern of sequential effects, whereby responses and response times depend on the detailed pattern of prior stimuli going back at least 5 trials. We show this pattern is well explained by simultaneous incremental learning of 2 simple statistics of the trial sequence: the base rate and the repetition rate. Both statistics are learned by the same basic associative mechanism, but they contribute different patterns of sequential effects because they entail different representations of the trial sequence. Subtler aspects of the data that are not explained by these 2 learning processes alone are explained by their interaction, via learning from joint error correction. Specifically, the cue-competition mechanism that has explained classic findings in animal learning (e.g., blocking) appears to operate on learning of sequence statistics. We also find that learning of the base rate and repetition rate are dissociated into response and stimulus processing, respectively, as indicated by event-related potentials, manipulations of stimulus discriminability, and reanalysis of past experiments that eliminated stimuli or prior responses. Thus, sequential effects in these tasks appear to be driven by learning the response base rate and the stimulus repetition rate. Connections are discussed between these findings and previous research attempting to separate stimulus- and response-based sequential effects, and research using sequential effects to determine mental representations. We conclude that sequential effects offer a powerful means for uncovering representations and learning mechanisms.

Reviewing erroneous information facilitates memory updating.

Reviewing information stored in memory will generally strengthen that information, so it seems reasonable that reviews should make it harder to replace the information in memory if it is later found to be erroneous. In Experiment 1, subjects learned three facts about each of 12 topics. On Day 2, the same facts were either reread, tested, or not reviewed; then the facts were "corrected" with new replacement facts. A test on the replacement facts given 1week later disclosed that both rereading and testing the to-be-replaced Day-1 facts enhanced memory for the Day-2 facts which supplanted them, although rereading (but not testing) the Day-1 facts also led to more intrusions of Day-1 facts on the final test. In Experiment 2, subjects were unexpectedly asked (in the final test) to recollect both original and replacement facts; old facts were often retrieved, especially when reviewed. It is suggested that review may promote development of a secondary retrieval route for the corrected information.

Learning where to look for a hidden target.

Survival depends on successfully foraging for food, for which evolution has selected diverse behaviors in different species. Humans forage not only for food, but also for information. We decide where to look over 170,000 times per day, approximately three times per wakeful second. The frequency of these saccadic eye movements belies the complexity underlying each individual choice. Experience factors into the choice of where to look and can be invoked to rapidly redirect gaze in a context and task-appropriate manner. However, remarkably little is known about how individuals learn to direct their gaze given the current context and task. We designed a task in which participants search a novel scene for a target whose location was drawn stochastically on each trial from a fixed prior distribution. The target was invisible on a blank screen, and the participants were rewarded when they fixated the hidden target location. In just a few trials, participants rapidly found the hidden targets by looking near previously rewarded locations and avoiding previously unrewarded locations. Learning trajectories were well characterized by a simple reinforcement-learning (RL) model that maintained and continually updated a reward map of locations. The RL model made further predictions concerning sensitivity to recent experience that were confirmed by the data. The asymptotic performance of both the participants and the RL model approached optimal performance characterized by an ideal-observer theory. These two complementary levels of explanation show how experience in a novel environment drives visual search in humans and may extend to other forms of search such as animal foraging.

The persistent impact of incidental experience.

As we perform daily activities--driving to work, unlocking the office door, or grabbing a coffee cup--our actions seem automatic and preprogrammed. Nonetheless, routine, well-practiced behavior is continually modulated by incidental experience: In repetitive experimental tasks, recent (~4) trials reliably influence performance and action choice. Psychological theories downplay the significance of sequential effects, explaining them as rapidly decaying perturbations of behavior, with no long-term consequences. We challenged this traditional perspective in two experiments designed to probe the impact of more distant experience, finding evidence for effects spanning up to a thousand intermediate trials. We present a normative theory in which these persistent effects reflect optimal adaptation to a dynamic environment exhibiting varying rates of change. The theory predicts a heavy-tailed decaying influence of past experience, consistent with our data, and suggests that individual incidental experiences are catalogued in a temporally extended memory utilized in order to optimize subsequent behavior.

When does fading enhance perceptual category learning?

Training that uses exaggerated versions of a stimulus discrimination (fading) has sometimes been found to enhance category learning, mostly in studies involving animals and impaired populations. However, little is known about whether and when fading facilitates learning for typical individuals. This issue was explored in 7 experiments. In Experiments 1 and 2, observers discriminated stimuli based on a single sensory continuum (time duration and line length, respectively). Adaptive fading dramatically improved performance in training (unsurprisingly) but did not enhance learning as assessed in a final test. The same was true for nonadaptive linear fading (Experiment 3). However, when variation in length (predicting category membership) was embedded among other (category-irrelevant) variation, fading dramatically enhanced not only performance in training but also learning as assessed in a final test (Experiments 4 and 5). Fading also helped learners to acquire a color saturation discrimination amid category-irrelevant variation in hue and brightness, although this learning proved transitory after feedback was withdrawn (Experiment 7). Theoretical implications are discussed, and we argue that fading should have practical utility in naturalistic category learning tasks, which involve extremely high dimensional stimuli and many irrelevant dimensions.

Sequential dependencies in driving.

The effect of recent experience on current behavior has been studied extensively in simple laboratory tasks. We explore the nature of sequential effects in the more naturalistic setting of automobile driving. Driving is a safety-critical task in which delayed response times may have severe consequences. Using a realistic driving simulator, we find significant sequential effects in pedal-press response times that depend on the history of recent stimuli and responses. Response times are slowed up to 100 ms in particular cases, a delay that has dangerous practical consequences. Further, we observe a significant number of history-related pedal misapplications, which have recently been noted as a cause for concern in the automotive safety community. By anticipating these consequences of sequential context, driver assistance systems could mitigate the effects of performance degradations and thus critically improve driver safety.

Object-based control of attention is sensitive to recent experience.

How is attention guided by past experience? In visual search, numerous studies have shown that recent trials influence responses to the current trial. Repeating features such as color, shape, or location of a target facilitates performance. Here we examine whether recent experience also modulates a more abstract dimension of attentional control, object-based and location-based control. Participants performed a cued target discrimination task with stimuli presented on 2 rectangles. Response times to targets appearing in an uncued location on a cued rectangle were faster than to targets on the uncued rectangle, demonstrating an object-based attentional benefit. We investigated the object-based benefit on the current trial contingent on the cue-target relationship on the previous trial. The object-based benefit was significant only when the cued object contained the target on the previous trial, not when the uncued object contained the target. This effect of recent experience was not due to either the repetition of spatial cue-target location or the repetition of the response, but to adaptation to contingencies in the environment. Our results suggest a unifying view of attentional control that spans the concrete dimensions of control (e.g., determining the relative importance of red vs. blue) to the abstract (determining the relative importance of objects vs. locations in space). Attention closely tracks the short time scale structure of the environment and automatically adapts to optimize performance to this structure.