Assessing computational reproducibility in Behavior Research Methods.

Psychological science has thrived thanks to new methods and innovative practices. Journals, including Behavior Research Methods (BRM), continue to support the dissemination and evaluation of research assets including data, software/hardware, statistical code, and databases of stimuli. However, such research assets rarely allow for computational reproducibility, meaning they are difficult to reuse. Therefore, in this preregistered report, we explore how BRM's authors and BRM structures shape the landscape of functional research assets. Our broad research questions concern: (1) How quickly methods and analytical techniques reported in BRM can be used and developed further by other scientists; (2) Whether functionality has improved following changes to BRM journal policy in support of computational reproducibility; (3) Whether we can disentangle such policy changes from changes in reproducibility over time. We randomly sampled equal numbers of papers (N = 204) published in BRM before and after the implementation of policy changes. Pairs of researchers recorded how long it took to ensure assets (data, software/hardware, statistical code, and materials) were fully operational. They also coded the completeness and reusability of the assets. While improvements were observed in all measures, only changes to completeness were altered significantly following the policy changes (d = .37). The effects varied between different types of research assets, with data sets from surveys/experiments showing the largest improvements in completeness and reusability. Perhaps more importantly, changes to policy do appear to have improved the life span of research products by reducing natural decline. We conclude with a discussion of how, in the future, research and policy might better support computational reproducibility within and beyond psychological science.

Opening Pandora's Box: Peeking inside Psychology's data sharing practices, and seven recommendations for change.

Open data-sharing is a valuable practice that ought to enhance the impact, reach, and transparency of a research project. While widely advocated by many researchers and mandated by some journals and funding agencies, little is known about detailed practices across psychological science. In a pre-registered study, we show that overall, few research papers directly link to available data in many, though not all, journals. Most importantly, even where open data can be identified, the majority of these lacked completeness and reusability-conclusions that closely mirror those reported outside of Psychology. Exploring the reasons behind these findings, we offer seven specific recommendations for engineering and incentivizing improved practices, so that the potential of open data can be better realized across psychology and social science more generally.

The facilitatory effect of negative feedback on the emergence of analogical reasoning abilities.

This paper focuses on the development of analogical reasoning abilities in 5- and 6-year-old children. Our particular interest relates to the way in which analogizing is influenced by the provision of task-based feedback coupled with a self-explanation requirement. Both feedback and self-explanation provide children with opportunities to engage in self-reflective thinking about the process of analogical reasoning. To examine the role of such metacognitive factors in analogical strategy development the reported study combined a proportional analogy paradigm with a small-scale microgenetic approach involving multiple testing sessions over a restricted time period. The key manipulation involved exposing participants either to the correct or incorrect analogy completions of another reasoner that they were then asked to explain. The data revealed that the development of an effective analogizing strategy embodying a 'relational shift' from superficial to relational responding was modulated by the feedback condition that the child was placed in, with a negative feedback intervention providing the greatest developmental benefit. We suggest that the value of negative feedback for the acquisition of analogical reasoning abilities derives from the way in which a self-reflective analysis of the reasons for erroneous responses sensitizes the child to a deeper understanding of how to make effective relational mappings.

The development of idiom comprehension: an investigation of semantic and contextual processing skills.

Two experiments compared 7- and 8-year-olds' and 9- and 10-year-olds' ability to use semantic analysis and inference from context to understand idioms. We used a multiple-choice task and manipulated whether the idioms were transparent or opaque, familiar or novel, and presented with or without a supportive story context. Performance was compared with that of adults (Experiment 1) and 11- and 12-year-olds (Experiment 2). The results broadly support Cacciari and Levorato's global elaboration model of figurative competence with a notable exception: Even the youngest children were able to use semantic analysis to derive the meanings of transparent idioms as well as being sensitive to meaning in context. The findings show that young children process language at both the small-grain phrase level and the discourse level to establish figurative meaning, and they demonstrate that the language processing skills that aid idiom comprehension, as well as idiom knowledge itself, are still not fully developed in 11- and 12-year-olds.

To get hold of the wrong end of the stick: reasons for poor idiom understanding in children with reading comprehension difficulties.

PURPOSE: The aim was to identify the source of idiom understanding difficulties in children with specific reading comprehension failure. METHOD: Two groups (ns = 15) of 9- to 10-year-olds participated. One group had age-appropriate word reading and reading comprehension; the other group had age-appropriate word reading but poor reading comprehension. Each child completed an independent assessment of semantic analysis skills and 2 multiple-choice assessments of idiom comprehension. In 1 assessment, idiomatic phrases were embedded in supportive story contexts; in the other assessment, they were presented out of context. Performance on transparent idioms (which are amenable to interpretation by semantic analysis) and opaque idioms (which can only be interpreted by inference from context if the meaning is not known) was compared. RESULTS: The groups demonstrated comparable semantic analysis skills and understanding of transparent idioms. Children with poor comprehension were impaired in the use of supportive context to aid their understanding of the opaque idioms. CONCLUSIONS: The study identifies poor inference from context as a source of the idiom understanding difficulties in children with poor reading comprehension; there was no evidence that poor semantic analysis skills contributed to their difficulties. Children with poor comprehension should be supported in the use of context to understand unfamiliar figurative language.