Ecological Validity and "Ecological Validity".

Egon Brunswik coined the term ecological validity to refer to the correlation between perceptual cues and the states and traits of a stimulus. Martin Orne adapted the term to refer to the generalization of experimental findings to the real world outside the laboratory. Both are legitimate uses of the term because the ecological validity of the cues in an experiment determines the ecological validity of the experiment itself.

Quo Vadis, Methodology? The Key Role of Manipulation Checks for Validity Control and Quality of Science.

The current debate about how to improve the quality of psychological science revolves, almost exclusively, around the subordinate level of statistical significance testing. In contrast, research design and strict theorizing, which are superordinate to statistics in the methods hierarchy, are sorely neglected. The present article is devoted to the key role assigned to manipulation checks (MCs) for scientific quality control. MCs not only afford a critical test of the premises of hypothesis testing but also (a) prompt clever research design and validity control, (b) carry over to refined theorizing, and (c) have important implications for other facets of methodology, such as replication science. On the basis of an analysis of the reality of MCs reported in current issues of the Journal of Personality and Social Psychology, we propose a future methodology for the post-p < .05 era that replaces scrutiny in significance testing with refined validity control and diagnostic research designs.

A Critical Review of Standardized Measures of Hypnotic Suggestibility.

The most well-established finding gleaned from decades of experimental hypnosis research is that individuals display marked variability in responsiveness to hypnotic suggestions. Insofar as this variability impacts both treatment outcome in therapeutic applications of hypnosis as well as responsiveness to suggestions in experimental contexts, it is imperative that clinicians and researchers use robust measures of hypnotic suggestibility. The current paper critically evaluates contemporary measures of hypnotic suggestibility. After reviewing the most widely used measures, we identify multiple properties of these instruments that result in the loss of valuable information, including binary scoring and single-trial sampling, and hinder their utility, such as the inclusion of suboptimal suggestion content. The scales are not well-suited for contemporary research questions and have outlived their usefulness. We conclude by outlining ways in which the measurement of hypnotic suggestibility can be advanced.

Practical Methodological Reform Needs Good Theory.

In the face of unreplicable results, statistical anomalies, and outright fraud, introspection and changes in the psychological sciences have taken root. Vibrant reform and metascience movements have emerged. These are exciting developments and may point toward practical improvements in the future. Yet there is nothing so practical as good theory. This article outlines aspects of reform and metascience in psychology that are ripe for an injection of theory, including a lot of excellent and overlooked theoretical work from different disciplines. I review established frameworks that model the process of scientific discovery, the types of scientific networks that we ought to aspire to, and the processes by which problematic norms and institutions might evolve, focusing especially on modeling from the philosophy of science and cultural evolution. We have unwittingly evolved a toxic scientific ecosystem; existing interdisciplinary theory may help us intelligently design a better one.

Experience with research paradigms relates to infants' direction of preference.

Interpreting and predicting direction of preference in infant research has been a thorny issue for decades. Several factors have been proposed to account for familiarity versus novelty preferences, including age, length of exposure, and task complexity. The current study explores an additional dimension: experience with the experimental paradigm. We reanalyzed the data from 4 experiments on artificial grammar learning in 12-month-old infants run using the head-turn preference procedure (HPP). Participants in these studies varied substantially in their number of laboratory visits. Results show that the number of HPP studies is related to direction of preference: Infants with limited experience with the HPP setting were more likely to show familiarity preferences than infants who had amassed more experience with this paradigm. This evidence has important implications for the interpretation of experimental results: Experience with a given method or, more broadly, with the laboratory environment may affect infants' patterns of preferences.

P-Curve Analysis of the Köhler Motivation Gain Effect in Exercise Settings: A Demonstration of a Novel Technique to Estimate Evidential Value Across Multiple Studies.

BACKGROUND: Practitioners and researchers may not always be able to adequately evaluate the evidential value of findings from a series of independent studies. This is partially due to the possibility of inflated effect size estimates for these findings as a result of researcher manipulation or selective reporting of analyses (i.e., p-hacking). In light of the possible overestimation of effect sizes in the literature, the p-curve analysis has been proposed as a worthwhile tool that may help identify bias across a series of studies focused on a single effect. The p-curve analysis provides a measure of the evidential value in the published literature and might highlight p-hacking practices. PURPOSE: Therefore, the purpose of this paper is to introduce the mechanics of the p-curve analysis to individuals researching phenomena in the psychosocial aspects of behavior and provide a substantive example of a p-curve analysis using findings from a series of studies examining a group dynamic motivation gain paradigm. METHODS: We performed a p-curve analysis on a sample of 13 studies that examined the Köhler motivation gain effect in exercise settings as a means to instruct readers how to conduct such an analysis on their own. RESULTS: The p-curve for studies examining the Köhler effect demonstrated evidential value and that this motivation effect is likely not a byproduct of p-hacking. The p-curve analysis is explained, as well as potential limitations of the analysis, interpretation of the results, and other uses where a p-curve analysis could be implemented.

Some researchers wear yellow pants, but even fewer participants read consent forms: Exploring and improving consent form reading in human subjects research.

Though consent forms include important information, those experienced with behavioral research often observe that participants do not carefully read consent forms. Three studies examined participants' reading of consent forms for in-person experiments. In each study, we inserted the phrase "some researchers wear yellow pants" into sections of the consent form and measured participants' reading of the form by testing their recall of the color yellow. In Study 1, we found that the majority of participants did not read consent forms thoroughly. This suggests that overall, participants sign consent forms that they have not read, confirming what has been observed anecdotally and documented in other research domains. Study 2 examined which sections of consent forms participants read and found that participants were more likely to read the first 2 sections of a consent form (procedure and risks) than later sections (benefits and anonymity and confidentiality). Given that rates of recall of the target phrase were under 70% even when the sentence was inserted into earlier sections of the form, we explored ways to improve participant reading in Study 3. Theorizing that the presence of a researcher may influence participants' retention of the form, we assigned participants to read the form with or without a researcher present. Results indicated that removing the researcher from the room while participants read the consent form decreased recall of the target phrase. Implications of these results and suggestions for future researchers are discussed. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The necessity of developing a two-stage qualification in ICD-11.

In the International Classification of Diseases, 10th Revision (ICD-10) two opposing principles are combined and mixed: atheoreticity that is necessary for the natural classification and commitment to nosology. Implementation of these principles requires a two-stage qualification. The first stage should be narrative detailed syndromological qualifications with identification of psychotic level of disorders. As for the second stage, the qualification should be nosological, based on complete clinical analysis, which is far from being possible to realize at once. ICD-10, specifically brought to nosological certainty, may remain the natural foundation for nosological qualification. Implementation of the syndromic qualification at the first stage will allow to consider nosological features of each syndrome at the second stage and to expand the list of criteria in different clusters. Such a suggestion opens the prospect for subsequent revisions of the ICD and allows to direct our efforts and those of practitioners to the unified channel, where the statistical goals would not be implemented at the expense of the research ones.

Mechanical Turk data collection in addiction research: utility, concerns and best practices.

AIMS: Amazon Mechanical Turk (MTurk) provides a crowdsourcing platform for the engagement of potential research participants with data collection instruments. This review (1) provides an introduction to the mechanics and validity of MTurk research; (2) gives examples of MTurk research; and (3) discusses current limitations and best practices in MTurk research. METHODS: We review four use cases of MTurk for research relevant to addictions: (1) the development of novel measures, (2) testing interventions, (3) the collection of longitudinal use data to determine the feasibility of longer-term studies of substance use and (4) the completion of large batteries of assessments to characterize the relationships between measured constructs. We review concerns with the platform, ways of mitigating these and important information to include when presenting findings. RESULTS: MTurk has proved to be a useful source of data for behavioral science more broadly, with specific applications to addiction science. However, it is still not appropriate for all use cases, such as population-level inference. To live up to the potential of highly transparent, reproducible science from MTurk, researchers should clearly report inclusion/exclusion criteria, data quality checks and reasons for excluding collected data, how and when data were collected and both targeted and actual participant compensation. CONCLUSIONS: Although on-line survey research is not a substitute for random sampling or clinical recruitment, the Mechanical Turk community of both participants and researchers has developed multiple tools to promote data quality, fairness and rigor. Overall, Mechanical Turk has provided a useful source of convenience samples despite its limitations and has demonstrated utility in the engagement of relevant groups for addiction science.

Gray (Literature) Matters: Evidence of Selective Hypothesis Reporting in Social Psychological Research.

Selective reporting practices (SRPs)-adding, dropping, or altering study elements when preparing reports for publication-are thought to increase false positives in scientific research. Yet analyses of SRPs have been limited to self-reports or analyses of pre-registered and published studies. To assess SRPs in social psychological research more broadly, we compared doctoral dissertations defended between 1999 and 2017 with the publications based on those dissertations. Selective reporting occurred in nearly 50% of studies. Fully supported dissertation hypotheses were 3 times more likely to be published than unsupported hypotheses, while unsupported hypotheses were nearly 4 times more likely to be dropped from publications. Few hypotheses were found to be altered or added post hoc. Dissertation studies with fewer supported hypotheses were more likely to remove participants or measures from publications. Selective hypothesis reporting and dropped measures significantly predicted greater hypothesis support in published studies, supporting concerns that SRPs may increase Type 1 error risk.

Belmont in Europe: A Mostly Indirect Influence.

This paper traces the reception of the Belmont Report in Europe and its influence on the development of European research ethics thinking and European research ethics systems. It is very difficult to trace a clear, linear reception history because it is difficult to disentangle the influence of the Report from the influence of concurrent developments, such as the 1975 revision of the World Medical Association Declaration of Helsinki and the requirement for research ethics review in the Vancouver Group's 1978 "Uniform Requirements for Manuscript Submission." The Report's insistence that the focus of research ethics should be the rights and interests of the individual research subject, and the use of an ethical framework and not ethical theory as the basis of analysis and justification of recommendations, were nevertheless very important for the development of research ethics. The divergence between Europe and the US in the governance of non-biomedical research can at least partly be explained by the absence of strong drivers for the introduction of research ethics committees outside of biomedicine in Europe, and by the ability of non-biomedical researchers to mobilize effectively against the introduction of such committees.

The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture.

Like many other areas of science, experimental psychology is affected by a "replication crisis" that is causing concern in many fields of research. Approaches to tackling this crisis include better training in statistical methods, greater transparency and openness, and changes to the incentives created by funding agencies, journals, and institutions. Here, I argue that if proposed solutions are to be effective, we also need to take into account human cognitive constraints that can distort all stages of the research process, including design and execution of experiments, analysis of data, and writing up findings for publication. I focus specifically on cognitive schemata in perception and memory, confirmation bias, systematic misunderstanding of statistics, and asymmetry in moral judgements of errors of commission and omission. Finally, I consider methods that may help mitigate the effect of cognitive constraints: better training, including use of simulations to overcome statistical misunderstanding; specific programmes directed at inoculating against cognitive biases; adoption of Registered Reports to encourage more critical reflection in planning studies; and using methods such as triangulation and "pre mortem" evaluation of study design to foster a culture of dialogue and criticism.