DataPipe: Born-open data collection for online experiments.

DataPipe ( https://pipe.jspsych.org ) is a tool that allows researchers to save data from a behavioral experiment directly to the Open Science Framework. Researchers can configure data storage options for an experiment on the DataPipe website and then use the DataPipe API to send data to the Open Science Framework from any Internet-connected experiment. DataPipe is free to use and open-source. This paper describes the design of DataPipe and how it can help researchers adopt the practice of born-open data collection.

Simulating behavior to help researchers build experiments.

Testing that an experiment works as intended is critical for identifying design problems and catching technical errors that could invalidate the results. Testing is also time-consuming because of the need to manually run the experiment. This makes testing the experiment costly for researchers, and therefore testing is less comprehensive than in other kinds of software development where tools to automate and speed up the testing process are widely used. In this paper, we describe an approach that substantially reduces the time required to test behavioral experiments: automated simulation of participant behavior. We describe how software that is used to build experiments can use information contained in the experiment's code to automatically generate plausible participant behavior. We demonstrate this through an implementation using jsPsych. We then describe four potential scenarios where automated simulation of participant behavior can improve the way researchers build experiments. Each scenario includes a demo and accompanying code. The full set of examples can be found at https://jspsych.github.io/simulation-examples/ .

Generalizability, transferability, and the practice-to-practice gap.

Emphasizing the predictive success and practical utility of psychological science is an admirable goal but it will require a substantive shift in how we design research. Applied research often assumes that findings are transferable to all practices, insensitive to variation between implementations. We describe efforts to quantify and close this practice-to-practice gap in education research.

A thousand studies for the price of one: Accelerating psychological science with Pushkin.

Half of the world's population has internet access. In principle, researchers are no longer limited to subjects they can recruit into the laboratory. Any study that can be run on a computer or mobile device can be run with nearly any demographic anywhere in the world, and in large numbers. This has allowed scientists to effectively run hundreds of experiments at once. Despite their transformative power, such studies remain rare for practical reasons: the need for sophisticated software, the difficulty of recruiting so many subjects, and a lack of research paradigms that make effective use of their large amounts of data, due to such realities as that they require sophisticated software in order to run effectively. We present Pushkin: an open-source platform for designing and conducting massive experiments over the internet. Pushkin allows for a wide range of behavioral paradigms, through integration with the intuitive and flexible jsPsych experiment engine. It also addresses the basic technical challenges associated with massive, worldwide studies, including auto-scaling, extensibility, machine-assisted experimental design, multisession studies, and data security.

Carving nature at its joints or cutting its effective loops? On the dangers of trying to disentangle intertwined mental processes.

Attention is often inextricably intertwined with perception, and it is deployed not only to spatial regions, but also to sensory dimensions, learned dimensions, and learned complex configurations. Firestone & Scholl's (F&S)'s tactic of isolating visual perceptual processes from attention and action has the negative consequence of neglecting interactions that are critically important for allowing people to perceive their world in efficient and useful ways.

Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task.

Behavioral researchers are increasingly using Web-based software such as JavaScript to conduct response time experiments. Although there has been some research on the accuracy and reliability of response time measurements collected using JavaScript, it remains unclear how well this method performs relative to standard laboratory software in psychologically relevant experimental manipulations. Here we present results from a visual search experiment in which we measured response time distributions with both Psychophysics Toolbox (PTB) and JavaScript. We developed a methodology that allowed us to simultaneously run the visual search experiment with both systems, interleaving trials between two independent computers, thus minimizing the effects of factors other than the experimental software. The response times measured by JavaScript were approximately 25 ms longer than those measured by PTB. However, we found no reliable difference in the variability of the distributions related to the software, and both software packages were equally sensitive to changes in the response times as a result of the experimental manipulations. We concluded that JavaScript is a suitable tool for measuring response times in behavioral research.

jsPsych: a JavaScript library for creating behavioral experiments in a Web browser.

Online experiments are growing in popularity, and the increasing sophistication of Web technology has made it possible to run complex behavioral experiments online using only a Web browser. Unlike with offline laboratory experiments, however, few tools exist to aid in the development of browser-based experiments. This makes the process of creating an experiment slow and challenging, particularly for researchers who lack a Web development background. This article introduces jsPsych, a JavaScript library for the development of Web-based experiments. jsPsych formalizes a way of describing experiments that is much simpler than writing the entire experiment from scratch. jsPsych then executes these descriptions automatically, handling the flow from one task to another. The jsPsych library is open-source and designed to be expanded by the research community. The project is available online at www.jspsych.org .

Using games to understand the mind.

Board, card or video games have been played by virtually every individual in the world. Games are popular because they are intuitive and fun. These distinctive qualities of games also make them ideal for studying the mind. By being intuitive, games provide a unique vantage point for understanding the inductive biases that support behaviour in more complex, ecological settings than traditional laboratory experiments. By being fun, games allow researchers to study new questions in cognition such as the meaning of 'play' and intrinsic motivation, while also supporting more extensive and diverse data collection by attracting many more participants. We describe the advantages and drawbacks of using games relative to standard laboratory-based experiments and lay out a set of recommendations on how to gain the most from using games to study cognition. We hope this Perspective will lead to a wider use of games as experimental paradigms, elevating the ecological validity, scale and robustness of research on the mind.

Rumors of Our Death .

Núñez and colleagues (2019) question whether cognitive science still exists "as a coherent academic field with a well-defined and cohesive interdisciplinary research program." This worry may be premature on two grounds. First, we are not convinced that the Lakatosian criterion of coalescence around a core framework is the best standard for judging whether a field is well-defined and productive. Second, although we acknowledge that cognitive science is not as visible as we would like, we doubt that this low profile accurately reflects the state of actual research and teaching programs based on the cognitive science approach.

A dissociation between engagement and learning: Enthusiastic instructions fail to reliably improve performance on a memory task.

Despite widespread assertions that enthusiasm is an important quality of effective teaching, empirical research on the effect of enthusiasm on learning and memory is mixed and largely inconclusive. To help resolve these inconsistencies, we conducted a carefully-controlled laboratory experiment, investigating whether enthusiastic instructions for a memory task would improve recall accuracy. Scripted videos, either enthusiastic or neutral, were used to manipulate the delivery of task instructions. We also manipulated the sequence of learning items, replicating the spacing effect, a known cognitive technique for memory improvement. Although spaced study reliably improved test performance, we found no reliable effect of enthusiasm on memory performance across two experiments. We did, however, find that enthusiastic instructions caused participants to respond to more item prompts, leaving fewer test questions blank, an outcome typically associated with increased task motivation. We find no support for the popular claim that enthusiastic instruction will improve learning, although it may still improve engagement. This dissociation between motivation and learning is discussed, as well as its implications for education and future research on student learning.

An In Vivo Study of Self-Regulated Study Sequencing in Introductory Psychology Courses.

Study sequence can have a profound influence on learning. In this study we investigated how students decide to sequence their study in a naturalistic context and whether their choices result in improved learning. In the study reported here, 2061 undergraduate students enrolled in an Introductory Psychology course completed an online homework tutorial on measures of central tendency, a topic relevant to an exam that counted towards their grades. One group of students was enabled to choose their own study sequence during the tutorial (Self-Regulated group), while the other group of students studied the same materials in sequences chosen by other students (Yoked group). Students who chose their sequence of study showed a clear tendency to block their study by concept, and this tendency was positively associated with subsequent exam performance. In the Yoked group, study sequence had no effect on exam performance. These results suggest that despite findings that blocked study is maladaptive when assigned by an experimenter, it may actually be adaptive when chosen by the learner in a naturalistic context.

Fitting perception in and to cognition.

Perceptual modules adapt at evolutionary, lifelong, and moment-to-moment temporal scales to better serve the informational needs of cognizers. Perceptual learning is a powerful way for an individual to become tuned to frequently recurring patterns in its specific local environment that are pertinent to its goals without requiring costly executive control resources to be deployed. Mechanisms like predictive coding, categorical perception, and action-informed vision allow our perceptual systems to interface well with cognition by generating perceptual outputs that are systematically guided by how they will be used. In classic conceptions of perceptual modules, people have access to the modules' outputs but no ability to adjust their internal workings. However, humans routinely and strategically alter their perceptual systems via training regimes that have predictable and specific outcomes. In fact, employing a combination of strategic and automatic devices for adapting perception is one of the most promising approaches to improving cognition.