P-Curve Analysis of the Köhler Motivation Gain Effect in Exercise Settings: A Demonstration of a Novel Technique to Estimate Evidential Value Across Multiple Studies.

BACKGROUND: Practitioners and researchers may not always be able to adequately evaluate the evidential value of findings from a series of independent studies. This is partially due to the possibility of inflated effect size estimates for these findings as a result of researcher manipulation or selective reporting of analyses (i.e., p-hacking). In light of the possible overestimation of effect sizes in the literature, the p-curve analysis has been proposed as a worthwhile tool that may help identify bias across a series of studies focused on a single effect. The p-curve analysis provides a measure of the evidential value in the published literature and might highlight p-hacking practices. PURPOSE: Therefore, the purpose of this paper is to introduce the mechanics of the p-curve analysis to individuals researching phenomena in the psychosocial aspects of behavior and provide a substantive example of a p-curve analysis using findings from a series of studies examining a group dynamic motivation gain paradigm. METHODS: We performed a p-curve analysis on a sample of 13 studies that examined the Köhler motivation gain effect in exercise settings as a means to instruct readers how to conduct such an analysis on their own. RESULTS: The p-curve for studies examining the Köhler effect demonstrated evidential value and that this motivation effect is likely not a byproduct of p-hacking. The p-curve analysis is explained, as well as potential limitations of the analysis, interpretation of the results, and other uses where a p-curve analysis could be implemented.

Experience with research paradigms relates to infants' direction of preference.

Interpreting and predicting direction of preference in infant research has been a thorny issue for decades. Several factors have been proposed to account for familiarity versus novelty preferences, including age, length of exposure, and task complexity. The current study explores an additional dimension: experience with the experimental paradigm. We reanalyzed the data from 4 experiments on artificial grammar learning in 12-month-old infants run using the head-turn preference procedure (HPP). Participants in these studies varied substantially in their number of laboratory visits. Results show that the number of HPP studies is related to direction of preference: Infants with limited experience with the HPP setting were more likely to show familiarity preferences than infants who had amassed more experience with this paradigm. This evidence has important implications for the interpretation of experimental results: Experience with a given method or, more broadly, with the laboratory environment may affect infants' patterns of preferences.

Belmont in Europe: A Mostly Indirect Influence.

This paper traces the reception of the Belmont Report in Europe and its influence on the development of European research ethics thinking and European research ethics systems. It is very difficult to trace a clear, linear reception history because it is difficult to disentangle the influence of the Report from the influence of concurrent developments, such as the 1975 revision of the World Medical Association Declaration of Helsinki and the requirement for research ethics review in the Vancouver Group's 1978 "Uniform Requirements for Manuscript Submission." The Report's insistence that the focus of research ethics should be the rights and interests of the individual research subject, and the use of an ethical framework and not ethical theory as the basis of analysis and justification of recommendations, were nevertheless very important for the development of research ethics. The divergence between Europe and the US in the governance of non-biomedical research can at least partly be explained by the absence of strong drivers for the introduction of research ethics committees outside of biomedicine in Europe, and by the ability of non-biomedical researchers to mobilize effectively against the introduction of such committees.

Rigor and reproducibility in rodent behavioral research.

Behavioral neuroscience research incorporates the identical high level of meticulous methodologies and exacting attention to detail as all other scientific disciplines. To achieve maximal rigor and reproducibility of findings, well-trained investigators employ a variety of established best practices. Here we explicate some of the requirements for rigorous experimental design and accurate data analysis in conducting mouse and rat behavioral tests. Novel object recognition is used as an example of a cognitive assay which has been conducted successfully with a range of methods, all based on common principles of appropriate procedures, controls, and statistics. Directors of Rodent Core facilities within Intellectual and Developmental Disabilities Research Centers contribute key aspects of their own novel object recognition protocols, offering insights into essential similarities and less-critical differences. Literature cited in this review article will lead the interested reader to source papers that provide step-by-step protocols which illustrate optimized methods for many standard rodent behavioral assays. Adhering to best practices in behavioral neuroscience will enhance the value of animal models for the multiple goals of understanding biological mechanisms, evaluating consequences of genetic mutations, and discovering efficacious therapeutics.

Combining speed and accuracy to control for speed-accuracy trade-offs(?).

In psychological experiments, participants are typically instructed to respond as fast as possible without sacrificing accuracy. How they interpret this instruction and, consequently, which speed-accuracy trade-off they choose might vary between experiments, between participants, and between conditions. Consequently, experimental effects can appear unpredictably in either RTs or error rates (i.e., accuracy). Even more problematic, spurious effects might emerge that are actually due only to differential speed-accuracy trade-offs. An often-suggested solution is the inverse efficiency score (IES; Townsend & Ashby, 1983), which combines speed and accuracy into a single score. Alternatives are the rate-correct score (RCS; Woltz & Was, 2006) and the linear-integrated speed-accuracy score (LISAS; Vandierendonck, 2017, 2018). We report analyses on simulated data generated with the standard diffusion model (Ratcliff, 1978) showing that IES, RCS, and LISAS put unequal weights on speed and accuracy, depending on the accuracy level, and that these measures are actually very sensitive to speed-accuracy trade-offs. These findings stand in contrast to a fourth alternative, the balanced integration score (BIS; Liesefeld, Fu, & Zimmer, 2015), which was devised to integrate speed and accuracy with equal weights. Although all of the measures maintain "real" effects, only BIS is relatively insensitive to speed-accuracy trade-offs.

The Thorny Relation Between Measurement Quality and Fit Index Cutoffs in Latent Variable Models.

Latent variable modeling is a popular and flexible statistical framework. Concomitant with fitting latent variable models is assessment of how well the theoretical model fits the observed data. Although firm cutoffs for these fit indexes are often cited, recent statistical proofs and simulations have shown that these fit indexes are highly susceptible to measurement quality. For instance, a root mean square error of approximation (RMSEA) value of 0.06 (conventionally thought to indicate good fit) can actually indicate poor fit with poor measurement quality (e.g., standardized factors loadings of around 0.40). Conversely, an RMSEA value of 0.20 (conventionally thought to indicate very poor fit) can indicate acceptable fit with very high measurement quality (standardized factor loadings around 0.90). Despite the wide-ranging effect on applications of latent variable models, the high level of technical detail involved with this phenomenon has curtailed the exposure of these important findings to empirical researchers who are employing these methods. This article briefly reviews these methodological studies in minimal technical detail and provides a demonstration to easily quantify the large influence measurement quality has on fit index values and how greatly the cutoffs would change if they were derived under an alternative level of measurement quality. Recommendations for best practice are also discussed.

Is automatic speech-to-text transcription ready for use in psychological experiments?

Verbal responses are a convenient and naturalistic way for participants to provide data in psychological experiments (Salzinger, The Journal of General Psychology, 61(1),65-94:1959). However, audio recordings of verbal responses typically require additional processing, such as transcribing the recordings into text, as compared with other behavioral response modalities (e.g., typed responses, button presses, etc.). Further, the transcription process is often tedious and time-intensive, requiring human listeners to manually examine each moment of recorded speech. Here we evaluate the performance of a state-of-the-art speech recognition algorithm (Halpern et al., 2016) in transcribing audio data into text during a list-learning experiment. We compare transcripts made by human annotators to the computer-generated transcripts. Both sets of transcripts matched to a high degree and exhibited similar statistical properties, in terms of the participants' recall performance and recall dynamics that the transcripts captured. This proof-of-concept study suggests that speech-to-text engines could provide a cheap, reliable, and rapid means of automatically transcribing speech data in psychological experiments. Further, our findings open the door for verbal response experiments that scale to thousands of participants (e.g., administered online), as well as a new generation of experiments that decode speech on the fly and adapt experimental parameters based on participants' prior responses.

Methods to detect low quality data and its implication for psychological research.

Web-based data collection methods such as Amazon's Mechanical Turk (AMT) are an appealing option to recruit participants quickly and cheaply for psychological research. While concerns regarding data quality have emerged with AMT, several studies have exhibited that data collected via AMT are as reliable as traditional college samples and are often more diverse and representative of noncollege populations. The development of methods to screen for low quality data, however, has been less explored. Omitting participants based on simple screening methods in isolation, such as response time or attention checks may not be adequate identification methods, with an inability to delineate between high or low effort participants. Additionally, problematic survey responses may arise from survey automation techniques such as survey bots or automated form fillers. The current project developed low quality data detection methods while overcoming previous screening limitations. Multiple checks were employed, such as page response times, distribution of survey responses, the number of utilized choices from a given range of scale options, click counts, and manipulation checks. This method was tested on a survey taken with an easily available plug-in survey bot, as well as compared to data collected by human participants providing both high effort and randomized, or low effort, answers. Identified cases can then be used as part of sensitivity analyses to warrant exclusion from further analyses. This algorithm can be a promising tool to identify low quality or automated data via AMT or other online data collection platforms.

Procura-PALavras (P-PAL): A Web-based interface for a new European Portuguese lexical database.

In this article, we present Procura-PALavras (P-PAL), a Web-based interface for a new European Portuguese (EP) lexical database. Based on a contemporary printed corpus of over 227 million words, P-PAL provides a broad range of word attributes and statistics, including several measures of word frequency (e.g., raw counts, per-million word frequency, logarithmic Zipf scale), morpho-syntactic information (e.g., parts of speech [PoSs], grammatical gender and number, dominant PoS, and frequency and relative frequency of the dominant PoS), as well as several lexical and sublexical orthographic (e.g., number of letters; consonant-vowel orthographic structure; density and frequency of orthographic neighbors; orthographic Levenshtein distance; orthographic uniqueness point; orthographic syllabification; and trigram, bigram, and letter type and token frequencies), and phonological measures (e.g., pronunciation, number of phonemes, stress, density and frequency of phonological neighbors, transposed and phonographic neighbors, syllabification, and biphone and phone type and token frequencies) for ~53,000 lemmatized and ~208,000 nonlemmatized EP word forms. To obtain these metrics, researchers can choose between two word queries in the application: (i) analyze words previously selected for specific attributes and/or lexical and sublexical characteristics, or (ii) generate word lists that meet word requirements defined by the user in the menu of analyses. For the measures it provides and the flexibility it allows, P-PAL will be a key resource to support research in all cognitive areas that use EP verbal stimuli. P-PAL is freely available at http://p-pal.di.uminho.pt/tools .

Is human classification by experienced untrained observers a gold standard in fixation detection?

Manual classification is still a common method to evaluate event detection algorithms. The procedure is often as follows: Two or three human coders and the algorithm classify a significant quantity of data. In the gold standard approach, deviations from the human classifications are considered to be due to mistakes of the algorithm. However, little is known about human classification in eye tracking. To what extent do the classifications from a larger group of human coders agree? Twelve experienced but untrained human coders classified fixations in 6 min of adult and infant eye-tracking data. When using the sample-based Cohen's kappa, the classifications of the humans agreed near perfectly. However, we found substantial differences between the classifications when we examined fixation duration and number of fixations. We hypothesized that the human coders applied different (implicit) thresholds and selection rules. Indeed, when spatially close fixations were merged, most of the classification differences disappeared. On the basis of the nature of these intercoder differences, we concluded that fixation classification by experienced untrained human coders is not a gold standard. To bridge the gap between agreement measures (e.g., Cohen's kappa) and eye movement parameters (fixation duration, number of fixations), we suggest the use of the event-based F1 score and two new measures: the relative timing offset (RTO) and the relative timing deviation (RTD).

Exploratory Factor Analysis With Small Samples and Missing Data.

Exploratory factor analysis (EFA) is an extremely popular method for determining the underlying factor structure for a set of variables. Due to its exploratory nature, EFA is notorious for being conducted with small sample sizes, and recent reviews of psychological research have reported that between 40% and 60% of applied studies have 200 or fewer observations. Recent methodological studies have addressed small size requirements for EFA models; however, these models have only considered complete data, which are the exception rather than the rule in psychology. Furthermore, the extant literature on missing data techniques with small samples is scant, and nearly all existing studies focus on topics that are not of primary interest to EFA models. Therefore, this article presents a simulation to assess the performance of various missing data techniques for EFA models with both small samples and missing data. Results show that deletion methods do not extract the proper number of factors and estimate the factor loadings with severe bias, even when data are missing completely at random. Predictive mean matching is the best method overall when considering extracting the correct number of factors and estimating factor loadings without bias, although 2-stage estimation was a close second.

Commentary 1: The Challenging Task of Addressing Power, Control, and Influence in Sibling Relationships After the First Two Decades of Life.

Extending the topic of power, control, and influence in the sibling relationship past the years covered in this issue is not an easy task because most adult siblings neither interact frequently nor live in geographic proximity, although there are considerable ethnic and social class variations. Thus, there are limited opportunities to observe how power is manifested, how siblings control each other and themselves to achieve their goals, nor how siblings influence each other. Comments in this commentary address the challenges presented when considering the trajectory of these dynamics in adult sibling relationships as well as the difficulties sibling dynamics present in carrying out tasks for which an egalitarian relationship is not ideal.

Marginally Significant Effects as Evidence for Hypotheses: Changing Attitudes Over Four Decades.

Some effects are statistically significant. Other effects do not reach the threshold of statistical significance and are sometimes described as "marginally significant" or as "approaching significance." Although the concept of marginal significance is widely deployed in academic psychology, there has been very little systematic examination of psychologists' attitudes toward these effects. Here, we report an observational study in which we investigated psychologists' attitudes concerning marginal significance by examining their language in over 1,500 articles published in top-tier cognitive, developmental, and social psychology journals. We observed a large change over the course of four decades in psychologists' tendency to describe a p value as marginally significant, and overall rates of use appear to differ across subfields. We discuss possible explanations for these findings, as well as their implications for psychological research.

Internet research in psychology.

Today the Internet plays a role in the lives of nearly 40% of the world's population, and it is becoming increasingly entwined in daily life. This growing presence is transforming psychological science in terms of the topics studied and the methods used. We provide an overview of the literature, considering three broad domains of research: translational (implementing traditional methods online; e.g., surveys), phenomenological (topics spawned or mediated by the Internet; e.g., cyberbullying), and novel (new ways to study existing topics; e.g., rumors). We discuss issues (e.g., sampling, ethics) that arise when doing research online and point to emerging opportunities (e.g., smartphone sensing). Psychological research on the Internet comes with new challenges, but the opportunities far outweigh the costs. By integrating the Internet, psychological research has the ability to reach large, diverse samples and collect data on actual behaviors, which will ultimately increase the impact of psychological research on society.

Thematic relatedness production norms for 100 object concepts.

Knowledge of thematic relations is an area of increased interest in semantic memory research because it is crucial to many cognitive processes. One methodological issue that researchers face is how to identify pairs of thematically related concepts that are well-established in semantic memory for most people. In this article, we review existing methods of assessing thematic relatedness and provide thematic relatedness production norming data for 100 object concepts. In addition, 1,174 related concept pairs obtained from the production norms were classified as reflecting one of the five subtypes of relations: attributive, argument, coordinate, locative, and temporal. The database and methodology will be useful for researchers interested in the effects of thematic knowledge on language processing, analogical reasoning, similarity judgments, and memory. These data will also benefit researchers interested in investigating potential processing differences among the five types of semantic relations.