Opening editorial 2022.

In this editorial, the author gives voice to three principles that he hopes will influence the articles that appear in American Psychologist: The scholarship should be rigorous, it should attend to both basic and applied considerations, and authors should be open and transparent in the reporting of how the research was conducted. The author says he would be remiss if he didn't also mention the importance of free and open inquiry. Progress in science depends on an atmosphere that permits-indeed, promotes-original thinking. A discipline, like psychology, that hopes to flourish by expanding the reach of its explanations must encourage divergent thinking and the thoughtful and respectful exchange of ideas while meeting criteria for excellent scholarship. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Aerobic exercise and neurocognitive performance: a meta-analytic review of randomized controlled trials.

OBJECTIVES: To assess the effects of aerobic exercise training on neurocognitive performance. Although the effects of exercise on neurocognition have been the subject of several previous reviews and meta-analyses, they have been hampered by methodological shortcomings and are now outdated as a result of the recent publication of several large-scale, randomized, controlled trials (RCTs). METHODS: We conducted a systematic literature review of RCTs examining the association between aerobic exercise training on neurocognitive performance between January 1966 and July 2009. Suitable studies were selected for inclusion according to the following criteria: randomized treatment allocation; mean age > or =18 years of age; duration of treatment >1 month; incorporated aerobic exercise components; supervised exercise training; the presence of a nonaerobic-exercise control group; and sufficient information to derive effect size data. RESULTS: Twenty-nine studies met inclusion criteria and were included in our analyses, representing data from 2049 participants and 234 effect sizes. Individuals randomly assigned to receive aerobic exercise training demonstrated modest improvements in attention and processing speed (g = 0.158; 95% confidence interval [CI]; 0.055-0.260; p = .003), executive function (g = 0.123; 95% CI, 0.021-0.225; p = .018), and memory (g = 0.128; 95% CI, 0.015-0.241; p = .026). CONCLUSIONS: Aerobic exercise training is associated with modest improvements in attention and processing speed, executive function, and memory, although the effects of exercise on working memory are less consistent. Rigorous RCTs are needed with larger samples, appropriate controls, and longer follow-up periods.

The relative benefits of meta-analysis conducted with individual participant data versus aggregated data.

The authors describe the relative benefits of conducting meta-analyses with (a) individual participant data (IPD) gathered from the constituent studies and (b) aggregated data (AD), or the group-level statistics (in particular, effect sizes) that appear in reports of a study's results. Given that both IPD and AD are equally available, meta-analysis of IPD is superior to meta-analysis of AD. IPD meta-analysis permits synthesists to perform subgroup analyses not conducted by the original collectors of the data, to check the data and analyses in the original studies, to add new information to the data sets, and to use different statistical methods. However, the cost of IPD meta-analysis and the lack of available IPD data sets suggest that the best strategy currently available is to use both approaches in a complementary fashion such that the first step in conducting an IPD meta-analysis would be to conduct an AD meta-analysis. Regardless of whether a meta-analysis is conducted with IPD or AD, synthesists must remain vigilant in how they interpret their results. They must avoid ecological fallacies, Simpson's paradox, and interpretation of synthesis-generated evidence as supporting causal inferences.

The effects of choice on intrinsic motivation and related outcomes: a meta-analysis of research findings.

A meta-analysis of 41 studies examined the effect of choice on intrinsic motivation and related outcomes in a variety of settings with both child and adult samples. Results indicated that providing choice enhanced intrinsic motivation, effort, task performance, and perceived competence, among other outcomes. Moderator tests revealed the effect of choice on intrinsic motivation was stronger (a) for instructionally irrelevant choices compared to choices made between activities, versions of a task, rewards, and instructionally relevant options, (b) when 2 to 4 successive choices were given, (c) when rewards were not given after the choice manipulation, (d) when participants given choice were compared to the most controlling forms of control groups, (e) for children compared to adults, (f) for designs that yoked choice and control conditions compared to matched designs in which choice was reduced or designs in which nonyoked, nonmatched controls were used, and (g) when the experiment was conducted in a laboratory embedded in a natural setting. Implications for future research and applications to real-world settings are discussed.

A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: the Study Design and Implementation Assessment Device (Study DIAD).

Assessments of studies meant to evaluate the effectiveness of interventions, programs, and policies can serve an important role in the interpretation of research results. However, evidence suggests that available quality assessment tools have poor measurement characteristics and can lead to opposing conclusions when applied to the same body of studies. These tools tend to (a) be insufficiently operational, (b) rely on arbitrary post-hoc decision rules, and (c) result in a single number to represent a multidimensional construct. In response to these limitations, a multilevel and hierarchical instrument was developed in consultation with a wide range of methodological and statistical experts. The instrument focuses on the operational details of studies and results in a profile of scores instead of a single score to represent study quality. A pilot test suggested that satisfactory between-judge agreement can be obtained using well-trained raters working in naturalistic conditions. Limitations of the instrument are discussed, but these are inherent in making decisions about study quality given incomplete reporting and in the absence of strong, contextually based information about the effects of design flaws on study outcomes.

Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report.

Following a review of extant reporting standards for scientific publication, and reviewing 10 years of experience since publication of the first set of reporting standards by the American Psychological Association (APA; APA Publications and Communications Board Working Group on Journal Article Reporting Standards, 2008), the APA Working Group on Quantitative Research Reporting Standards recommended some modifications to the original standards. Examples of modifications include division of hypotheses, analyses, and conclusions into 3 groupings (primary, secondary, and exploratory) and some changes to the section on meta-analysis. Several new modules are included that report standards for observational studies, clinical trials, longitudinal studies, replication studies, and N-of-1 studies. In addition, standards for analytic methods with unique characteristics and output (structural equation modeling and Bayesian analysis) are included. These proposals were accepted by the Publications and Communications Board of APA and supersede the standards included in the 6th edition of the Publication Manual of the American Psychological Association (APA, 2010). (PsycINFO Database Record

Intent of reporting standards: Reply to Rossiter (2018).

In this reply to Rossiter (2018), we note that the goal of developing Journal Article Reporting Standards has been to specify the kinds of information that should be provided to the readers of scientific articles in order to allow maximal understanding of the work being reported-in the case of psychometrics, information that demonstrates the underlying adequacy of the measures used in the research being reported. Although we illustrate some kinds of items that might be utilized to make these demonstrations, the illustrations are not proscriptive. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate.

A complete description of the literature search, including the criteria used for the inclusion of reports after they have been located, used in a research synthesis or meta-analysis is critical if subsequent researchers are to accurately evaluate and reproduce a synthesis' methods and results. Based on previous guidelines and new suggestions, we present a set of focused and detailed standards for reporting the methods used in a literature search. The guidelines cover five search strategies: reference database searches, journal and bibliography searches, searches of the reference lists of reports, citation searches, and direct contact searches. First, we bring together all the unique recommendations made in existing guidelines for research synthesis. Second, we identify gaps in reporting standards for search strategies. Third, we address these gaps by providing new reporting recommendations. Our hope is to facilitate successful evaluation and replication of research synthesis results.

Cues to deception.

Do people behave differently when they are lying compared with when they are telling the truth? The combined results of 1,338 estimates of 158 cues to deception are reported. Results show that in some ways, liars are less forthcoming than truth tellers, and they tell less compelling tales. They also make a more negative impression and are more tense. Their stories include fewer ordinary imperfections and unusual contents. However, many behaviors showed no discernible links, or only weak links, to deceit. Cues to deception were more pronounced when people were motivated to succeed, especially when the motivations were identity relevant rather than monetary or material. Cues to deception were also stronger when lies were about transgressions.

A brief history of research synthesis.

Science is supposed to be cumulative, but scientists only rarely cumulate evidence scientifically. This means that users of research evidence have to cope with a plethora of reports of individual studies with no systematic attempt made to present new results in the context of similar studies. Although the need to synthesize research evidence has been recognized for well over two centuries, explicit methods for this form of research were not developed until the 20th century. The development of methods to reduce statistical imprecision using quantitative synthesis (meta-analysis) preceded the development of methods to reduce biases, the latter only beginning to receive proper attention during the last quarter of the 20th century. In this article, the authors identify some of the trends and highlights in this history, to which researchers in the physical, natural, and social sciences have all contributed, and speculate briefly about the "future history" of research synthesis.

The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship.

In the past two decades, a new form of scholarship has appeared in which researchers present an overview of previously conducted research syntheses on the same topic. In these efforts, research syntheses are the principal units of evidence. Overviews of reviews introduce unique problems that require unique solutions. This article describes what methods overviewers have developed or have adopted from other forms of scholarship. These methods concern how to (a) define the broader problem space of an overview, (b) conduct literature searches that specifically look for research syntheses, (c) address the overlap in evidence in related reviews, (d) evaluate the quality of both primary research and research syntheses, (e) integrate the outcomes of research syntheses, especially when they produce discordant results, (f) conduct a second-order meta-analysis, and (g) present findings. The limitations of overviews are also discussed, especially with regard to the age of the included evidence.

A method for evaluating research syntheses: The quality, conclusions, and consensus of 12 syntheses of the effects of after-school programs.

Like all forms of empirical inquiry, research syntheses can be carried out in ways that lead to more or less valid inferences about the phenomenon under study. This synthesis of syntheses (a) examined the methods employed in the syntheses of the effects of after-school programs (ASPs) and determined how closely they conformed to what is defined as best practice for research synthesis, (b) compared the inferences drawn from the ASP research literature by each synthesis with the inferences that plausibly could be made from the data they covered, and (c) determined the points of consistency across the syntheses with regard to both potentially valid and potentially invalid conclusions. It was found that the 12 syntheses used highly divergent methods, varying in problem definitions, search strategies, inclusion criteria for individual studies, and techniques for drawing conclusions about the cumulative evidence. Copyright © 2010 John Wiley & Sons, Ltd.

Interventions to increase physical activity among aging adults: a meta-analysis.

OBJECTIVES: This review applied meta-analytic procedures to integrate primary research findings that test interventions to increase activity among aging adults. METHODS: We performed extensive literature searching strategies and located published and unpublished intervention studies that measured the activity behavior of at least five participants with a mean age of 60 years or greater. Primary study results were coded, and meta-analytic procedures were conducted. RESULTS: The overall effect size, weighted by sample size, was d(w) = .26 +/- .05. Effect sizes were larger when interventions targeted only activity behavior, excluded general health education, incorporated self-monitoring, used center-based exercise, recommended moderate intensity activity, were delivered in groups, used intense contact between interventionists and participants, and targeted patient populations. Effect sizes were larger for studies that measured exercise duration and studies with a time interval of less than 90 days between intervention and behavior measurement. CONCLUSIONS: These findings suggest that group-delivered interventions should encourage moderate activity, incorporate self-monitoring, target only activity, and encourage center-based activity. Findings also suggest that patient populations may be especially receptive to activity interventions. Primary research testing interventions in randomized trials to confirm causal relationships would be constructive.

Grey literature in meta-analyses.

BACKGROUND: In meta-analysis, researchers combine the results of individual studies to arrive at cumulative conclusions. Meta-analysts sometimes include "grey literature" in their evidential base, which includes unpublished studies and studies published outside widely available journals. Because grey literature is a source of data that might not employ peer review, critics have questioned the validity of its data and the results of meta-analyses that include it. OBJECTIVE: To examine evidence regarding whether grey literature should be included in meta-analyses and strategies to manage grey literature in quantitative synthesis. METHODS: This article reviews evidence on whether the results of studies published in peer-reviewed journals are representative of results from broader samplings of research on a topic as a rationale for inclusion of grey literature. Strategies to enhance access to grey literature are addressed. RESULTS: The most consistent and robust difference between published and grey literature is that published research is more likely to contain results that are statistically significant. Effect size estimates of published research are about one-third larger than those of unpublished studies. Unfunded and small sample studies are less likely to be published. Yet, importantly, methodological rigor does not differ between published and grey literature. CONCLUSIONS: Meta-analyses that exclude grey literature likely (a) over-represent studies with statistically significant findings, (b) inflate effect size estimates, and (c) provide less precise effect size estimates than meta-analyses including grey literature. Meta-analyses should include grey literature to fully reflect the existing evidential base and should assess the impact of methodological variations through moderator analysis.

Effectiveness of mentoring programs for youth: a meta-analytic review.

We used meta-analysis to review 55 evaluations of the effects of mentoring programs on youth. Overall, findings provide evidence of only a modest or small benefit of program participation for the average youth. Program effects are enhanced significantly, however, when greater numbers of both theory-based and empirically based "best practices" are utilized and when strong relationships are formed between mentors and youth. Youth from backgrounds of environmental risk and disadvantage appear most likely to benefit from participation in mentoring programs. Outcomes for youth at-risk due to personal vulnerabilities have varied substantially in relation to program characteristics, with a noteworthy potential evident for poorly implemented programs to actually have an adverse effect on such youth. Recommendations include greater adherence to guidelines for the design and implementation of effective mentoring programs as well as more in-depth assessment of relationship and contextual factors in the evaluation of programs.