A framework for rigorous evaluation of human performance in human and machine learning comparison studies.

Rigorous comparisons of human and machine learning algorithm performance on the same task help to support accurate claims about algorithm success rates and advances understanding of their performance relative to that of human performers. In turn, these comparisons are critical for supporting advances in artificial intelligence. However, the machine learning community has lacked a standardized, consensus framework for performing the evaluations of human performance necessary for comparison. We demonstrate common pitfalls in a designing the human performance evaluation and propose a framework for the evaluation of human performance, illustrating guiding principles for a successful comparison. These principles are first, to design the human evaluation with an understanding of the differences between human and algorithm cognition; second, to match trials between human participants and the algorithm evaluation, and third, to employ best practices for psychology research studies, such as the collection and analysis of supplementary and subjective data and adhering to ethical review protocols. We demonstrate our framework's utility for designing a study to evaluate human performance on a one-shot learning task. Adoption of this common framework may provide a standard approach to evaluate algorithm performance and aid in the reproducibility of comparisons between human and machine learning algorithm performance.

Benchmarking Human Performance for Visual Search of Aerial Images.

Aerial images are frequently used in geospatial analysis to inform responses to crises and disasters but can pose unique challenges for visual search when they contain low resolution, degraded information about color, and small object sizes. Aerial image analysis is often performed by humans, but machine learning approaches are being developed to complement manual analysis. To date, however, relatively little work has explored how humans perform visual search on these tasks, and understanding this could ultimately help enable human-machine teaming. We designed a set of studies to understand what features of an aerial image make visual search difficult for humans and what strategies humans use when performing these tasks. Across two experiments, we tested human performance on a counting task with a series of aerial images and examined the influence of features such as target size, location, color, clarity, and number of targets on accuracy and search strategies. Both experiments presented trials consisting of an aerial satellite image; participants were asked to find all instances of a search template in the image. Target size was consistently a significant predictor of performance, influencing not only accuracy of selections but the order in which participants selected target instances in the trial. Experiment 2 demonstrated that the clarity of the target instance and the match between the color of the search template and the color of the target instance also predicted accuracy. Furthermore, color also predicted the order of selecting instances in the trial. These experiments establish not only a benchmark of typical human performance on visual search of aerial images but also identify several features that can influence the task difficulty level for humans. These results have implications for understanding human visual search on real-world tasks and when humans may benefit from automated approaches.

Working memory plasticity and aging.

The present research explores how the trajectory of learning on a working memory task changes throughout the life span, and whether gains in working memory performance are exclusively a question of initial working memory capacity (WMC) or whether age exerts an independent effect. In a large, cross-sectional study of younger, middle-aged, and older adults, we examined learning on a widely used working memory task-the dual n-back task-over 20 sessions of practice. We found that, while all age groups improved on the task, older adults demonstrated less improvement on the task, and also reached a lower asymptotic maximum performance than younger adults. After controlling for initial WMC, we found that age exerted independent effects on training gains and asymptotic performance; older adults tended to improve less and reached lower levels of performance than younger adults. The difference between younger and older adults' rates of learning depended in part on initial WMC. These results suggest that age-related effects on working memory include not only effects on capacity, but also plasticity and the ability to improve on a task. (PsycINFO Database Record

Explaining the alluring influence of neuroscience information on scientific reasoning.

Previous studies have investigated the influence of neuroscience information or images on ratings of scientific evidence quality but have yielded mixed results. We examined the influence of neuroscience information on evaluations of flawed scientific studies after taking into account individual differences in scientific reasoning skills, thinking dispositions, and prior beliefs about a claim. We found that neuroscience information, even though irrelevant, made people believe they had a better understanding of the mechanism underlying a behavioral phenomenon. Neuroscience information had a smaller effect on ratings of article quality and scientist quality. Our study suggests that neuroscience information may provide an illusion of explanatory depth.

Task difficulty modulates the impact of emotional stimuli on neural response in cognitive-control regions.

Both heightened reactivity to emotional stimuli and impaired cognitive control are key aspects of depression, anxiety, and addiction. But the impact of emotion on cognitive-control processes, and the factors that modulate this impact, are still not well understood. We examined the effects of threat and reward distracters on the neural correlates of cognitive control using functional MRI (fMRI) and the Multi-Source Interference Task (MSIT). Behaviorally, subjects were slower and less accurate on the more demanding incongruent trials compared to the easier congruent trials. In addition, both threat and reward distracters significantly impaired the speed of responding on incongruent trials relative to the no-distracter condition. At the neural level, we used the incongruent - congruent contrast to functionally define four cognitive-control regions of interest (ROIs): anterior cingulate cortex (ACC), left and right inferior frontal gyrus (IFG)/insula, and right dorsolateral prefrontal cortex (DLPFC). A repeated-measures analysis of variance on the extracted contrast values in these ROIs indicated a significant interaction of stimulus salience and task difficulty on the neural response in cognitive-control regions. Specifically, threat distracters significantly decreased the response in cognitive-control regions on incongruent trials, whereas they significantly increased that response on congruent trials, relative to the no-distracter condition. Exploratory analyses of the amygdala response showed a similar interaction of stimulus salience and task difficulty: threat distracters significantly decreased the amygdala response only on incongruent trials. Overall, our results suggest that the impact of emotional distracters on the neural response in cognitive-control regions as well as in the amygdala is modulated by task difficulty, and add to our understanding of the factors that determine whether emotion enhances or impairs cognition.