Meta-analysis in a digitalized world: A step-by-step primer.

In recent years, much research and many data sources have become digital. Some advantages of digital or Internet-based research, compared to traditional lab research (e.g., comprehensive data collection and storage, availability of data) are ideal for an improved meta-analyses approach.In the meantime, in meta-analyses research, different types of meta-analyses have been developed to provide research syntheses with accurate quantitative estimations. Due to its rich and unique palette of corrections, we recommend to using the Schmidt and Hunter approach for meta-analyses in a digitalized world. Our primer shows in a step-by-step fashion how to conduct a high quality meta-analysis considering digital data and highlights the most obvious pitfalls (e.g., using only a bare-bones meta-analysis, no data comparison) not only in aggregation of the data, but also in the literature search and coding procedure which are essential steps in any meta-analysis. Thus, this primer of meta-analyses is especially suited for a situation where much of future research is headed to: digital research. To map Internet-based research and to reveal any research gap, we further synthesize meta-analyses on Internet-based research (15 articles containing 24 different meta-analyses, on 745 studies, with 1,601 effect sizes), resulting in the first mega meta-analysis of the field. We found a lack of individual participant data (e.g., age and nationality). Hence, we provide a primer for high-quality meta-analyses and mega meta-analyses that applies to much of coming research and also basic hands-on knowledge to conduct or judge the quality of a meta-analyses in a digitalized world.

Illustrating psychometric tests, scales, and constructs: An R package for Item Pool Visualization.

Researchers assessing psychological constructs have to understand and choose between several competing measures. Item Pool Visualization (IPV, Dantlgraber et al., 2019) was developed to offer a systematic and detailed portrayal of the actual content and internal balance of competing measures. To enable the use of IPV, we developed and present here the IPV R package. Its aim is to allow researchers to add IPV to their repertoire with minimal effort. Creating IPV charts from raw data requires two simple function calls, because the package streamlines model specification, model estimation, and chart creation. It improves IPV conceptually by introducing the aggregate center distance and the item overview chart. It provides many customization options and generates high-quality, vector-based PDF output. The workflow of the package is explained using a reproducible open data example from a personality assessment.

Geofencing in location-based behavioral research: Methodology, challenges, and implementation.

This manuscript presents a novel geofencing method in behavioral research. Geofencing, built upon geolocation technology, constitutes virtual fences around specific locations. Every time a participant crosses the virtual border around the geofenced area, an event can be triggered on a smartphone, e.g., the participant may be asked to complete a survey. The geofencing method can alleviate the problems of constant location tracking, such as recording sensitive geolocation information and battery drain. In scenarios where locations for geofencing are determined by participants (e.g., home, workplace), no location data need to be transferred to the researcher, so this method can ensure privacy and anonymity. Given the widespread use of smartphones and mobile Internet, geofencing has become a feasible tool in studying human behavior and cognition outside of the laboratory. The method can help advance theoretical and applied psychological science at a new frontier of context-aware research. At the same time, there is a lack of guidance on how and when geofencing can be applied in research. This manuscript aims to fill the gap and ease the adoption of the geofencing method. We describe the current challenges and implementations in geofencing and present three empirical studies in which we evaluated the geofencing method using the Samply application, a tool for mobile experience sampling research. The studies show that sensitivity and precision of geofencing were affected by the type of event, location radius, environment, operating system, and user behavior. Potential implications and recommendations for behavioral research are discussed.

[Psychometric Properties of a German Translation of the Parental Bonding Instrument]. / Psychometrische Kennwerte einer deutschen Übersetzung des Parental Bonding Instrument.

Parenting behavior affects a child's development as well as the etiology and treatment of mental disorders. The Parental Bonding Instrument (PBI; Parker, Tupling & Brown, 1979) is a well-known measurement tool to retrospectively assess parenting styles. Yet, no sufficiently validated German version exists to date. Therefore, we developed an updated translation of the German PBI version (PBI-dt) and analyzed its psychometric properties in an online survey based on a sample of n=791 German-speaking participants with a focus on item and reliability characteristics, construct and criterion validity as well as factorial structure of the PBI-dt.Our results indicated good item characteristics and reliability (α=0.86-0.95). Correlations between PBI and CTQ-SF (Childhood Trauma Questionnaire Short Form) scales were in line with the literature. Significant differences in the reported parenting style were found between people with and without mental illness as well as between normal-weight and overweight people. These results indicated the presence of good construct and criterion validity. Confirmatory factor analyses indicated an acceptable model fit for all fit indices in the original 2-factor model of Parker et al. (1979) as well as in the 3-factor model with the scales CareCareCareCareCareCare, Discouragement of behavioral freedomDiscouragement of behavioral freedomDiscouragement of behavioral freedomDiscouragement of behavioral freedomDiscouragement of behavioral freedomDiscouragement of behavioral freedom and Denial of psychological autonomyDenial of psychological autonomyDenial of psychological autonomyDenial of psychological autonomyDenial of psychological autonomyDenial of psychological autonomy. A 3-factor structure provided additional information, e. g., a better differentiation between normal and overweight people. Hence, this German translation of the PBI has good psychometric properties and is a reliable measuring instrument.

Smartphone sensor accuracy varies from device to device in mobile research: The case of spatial orientation.

Smartphone usage is increasing around the globe-in daily life and as a research device in behavioral science. Smartphones offer the possibility to gather longitudinal data at little cost to researchers and participants. They provide the option to verify self-report data with data from sensors built into most smartphones. How accurate this sensor data is when gathered via different smartphone devices, e.g., in a typical experience sampling framework, has not been investigated systematically. With the present study, we investigated the accuracy of orientation data about the spatial position of smartphones via a newly invented measurement device, the RollPitcher. Objective status of pitch (vertical orientation) and roll (horizontal orientation) of the smartphone was compared to data gathered from the sensors via web browsers and native apps. Bayesian ANOVAs confirmed that the deviations in pitch and roll differed between smartphone models, with mean inaccuracies per device of up to 2.1° and 6.6°, respectively. The inaccuracies for measurements of roll were higher than for pitch, d = .28, p < .001. Our results confirm the presence of heterogeneities when gathering orientation data from different smartphone devices. In most cases, measurement via a web browser was identical to measurement via a native app, but this was not true for all smartphone devices. As a solution to lack of sensor accuracy, we recommend the development and implementation of a coherent research framework and also discuss the implications of the heterogeneities in orientation data for different research designs.

Samply: A user-friendly smartphone app and web-based means of scheduling and sending mobile notifications for experience-sampling research.

Undertaking an experience-sampling study via smartphones is complex. Scheduling and sending mobile notifications often requires the use of proprietary software that imposes limits on participants' operating systems (whether iOS or Android) or the types of questions that can be asked via the application. We have developed an open-source platform-Samply-which overcomes these limitations. Researchers can access the entire interface via a browser, manage studies, schedule and send notifications linking to online surveys or experiments created in any Internet-based service or software, and monitor participants' responses-all without the coding skills usually needed to program a native mobile application. Participants can download the Samply Research mobile application for free from Google Play or the App Store, join a specific study, receive notifications and web links to surveys or experiments, and track their involvement. The mobile application leverages the power of the React Native JavaScript library, which allows it to be rendered in the native code of Android and iOS mobile operating systems. We describe Samply, provide a step-by-step example of conducting an experience-sampling study, and present the results of two validation studies. Study 1 demonstrates how we improved the website's usability for researchers. Study 2 validates the mobile application's data recording ability by analyzing a survey's participation rate. The application's possible limitations and how mobile device settings might affect its reliability are discussed.

Individual differences influence two-digit number processing, but not their analog magnitude processing: a large-scale online study.

Symbolic magnitude comparison is one of the most well-studied cognitive processes in research on numerical cognition. However, while the cognitive mechanisms of symbolic magnitude processing have been intensively studied, previous studies have paid less attention to individual differences influencing symbolic magnitude comparison. Employing a two-digit number comparison task in an online setting, we replicated previous effects, including the distance effect, the unit-decade compatibility effect, and the effect of cognitive control on the adaptation to filler items, in a large-scale study in 452 adults. Additionally, we observed that the most influential individual differences were participants' first language, time spent playing computer games and gender, followed by reported alcohol consumption, age and mathematical ability. Participants who used a first language with a left-to-right reading/writing direction were faster than those who read and wrote in the right-to-left direction. Reported playing time for computer games was correlated with faster reaction times. Female participants showed slower reaction times and a larger unit-decade compatibility effect than male participants. Participants who reported never consuming alcohol showed overall slower response times than others. Older participants were slower, but more accurate. Finally, higher grades in mathematics were associated with faster reaction times. We conclude that typical experiments on numerical cognition that employ a keyboard as an input device can also be run in an online setting. Moreover, while individual differences have no influence on domain-specific magnitude processing-apart from age, which increases the decade distance effect-they generally influence performance on a two-digit number comparison task.

Best practices: Two Web-browser-based methods for stimulus presentation in behavioral experiments with high-resolution timing requirements.

The Web is a prominent platform for behavioral experiments, for many reasons (relative simplicity, ubiquity, and accessibility, among others). Over the last few years, many behavioral and social scientists have conducted Internet-based experiments using standard web technologies, both in native JavaScript and using research-oriented frameworks. At the same time, vendors of widely used web browsers have been working hard to improve the performance of their software. However, the goals of browser vendors do not always coincide with behavioral researchers' needs. Whereas vendors want high-performance browsers to respond almost instantly and to trade off accuracy for speed, researchers have the opposite trade-off goal, wanting their browser-based experiments to exactly match the experimental design and procedure. In this article, we review and test some of the best practices suggested by web-browser vendors, based on the features provided by new web standards, in order to optimize animations for browser-based behavioral experiments with high-resolution timing requirements. Using specialized hardware, we conducted four studies to determine the accuracy and precision of two different methods. The results using CSS animations in web browsers (Method 1) with GPU acceleration turned off showed biases that depend on the combination of browser and operating system. The results of tests on the latest versions of GPU-accelerated web browsers showed no frame loss in CSS animations. The same happened in many, but not all, of the tests conducted using requestAnimationFrame (Method 2) instead of CSS animations. Unbeknownst to many researchers, vendors of web browsers implement complex technologies that result in reduced quality of timing. Therefore, behavioral researchers interested in timing-dependent procedures should be cautious when developing browser-based experiments and should test the accuracy and precision of the whole experimental setup (web application, web browser, operating system, and hardware).

The SNARC and MARC effects measured online: Large-scale assessment methods in flexible cognitive effects.

The Spatial-Numerical Association of Response Codes (SNARC) effect (i.e., faster reactions to small/large numbers on the left-/right-hand side) is usually observed along with the linguistic Markedness of Response Codes (MARC) effect-that is, faster left-/right-hand responses to odd/even numbers. The SNARC effect is one of the most thoroughly investigated phenomena in numerical cognition. However, almost all SNARC and MARC studies to date were conducted with sample sizes smaller than 100. Here we report on a study with 1,156 participants from various linguistic and cultural backgrounds performing a typical parity judgment task. We investigated whether (1) the SNARC and MARC effects can be observed in an online setup, (2) the properties of these effects observed online are similar to those observed in laboratory setups, (3) the effects are reliable, and (4) they are valid. We found robust SNARC and MARC effects. Their magnitude and reliabilities were comparable to values previously reported in in-lab studies. Furthermore, we reproduced commonly observed validity correlations of the SNARC and MARC effects. Namely, SNARC and MARC correlated with mean reaction times and intraindividual variability in reaction times. Additionally, we found interindividual differences in the SNARC and MARC effects (e.g., finger-counting routines for the SNARC and handedness for the MARC). Large-scale testing via web-based data acquisition not only produces SNARC and MARC effects and validity correlations similar to those from small, in-lab studies, but also reveals substantial insights with regard to interindividual differences that usually cannot be revealed in the offline laboratory, due to power considerations.

Questions on honest responding.

This article presents a new method for reducing socially desirable responding in Internet self-reports of desirable and undesirable behavior. The method is based on moving the request for honest responding, often included in the introduction to surveys, to the questioning phase of the survey. Over a quarter of Internet survey participants do not read survey instructions, and therefore, instead of asking respondents to answer honestly, they were asked whether they responded honestly. Posing the honesty message in the form of questions on honest responding draws attention to the message, increases the processing of it, and puts subsequent questions in context with the questions on honest responding. In three studies (nStudy I = 475, nStudy II = 1,015, nStudy III = 899), we tested whether presenting the questions on honest responding before questions on desirable and undesirable behavior could increase the honesty of responses, under the assumption that less attribution of desirable behavior and/or admitting to more undesirable behavior could be taken to indicate more honest responses. In all studies the participants who were presented with the questions on honest responding before questions on the target behavior produced, on average, significantly less socially desirable responses, though the effect sizes were small in all cases (Cohen's d ranging between 0.02 and 0.28 for single items, and from 0.17 to 0.34 for sum scores). The overall findings and the possible mechanisms behind the influence of the questions concerning honest responding on subsequent questions are discussed, and suggestions are made for future research.

Can smartphones be used to bring computer-based tasks from the lab to the field? A mobile experience-sampling method study about the pace of life.

Researchers are increasingly using smartphones to collect scientific data. To date, most smartphone studies have collected questionnaire data or data from the built-in sensors. So far, few studies have analyzed whether smartphones can also be used to conduct computer-based tasks (CBTs). Using a mobile experience-sampling method study and a computer-based tapping task as examples (N = 246; twice a day for three weeks, 6,000+ measurements), we analyzed how well smartphones can be used to conduct a CBT. We assessed methodological aspects such as potential technologically induced problems, dropout, task noncompliance, and the accuracy of millisecond measurements. Overall, we found few problems: Dropout rate was low, and the time measurements were very accurate. Nevertheless, particularly at the beginning of the study, some participants did not comply with the task instructions, probably because they did not read the instructions before beginning the task. To summarize, the results suggest that smartphones can be used to transfer CBTs from the lab to the field, and that real-world variations across device manufacturers, OS types, and CPU load conditions did not substantially distort the results.

The changing psychology of culture in German-speaking countries: A Google Ngram study.

This article provides evidence for the long-term affiliation between ecological and cultural changes in German-speaking countries, based on the assumptions derived from social change and human development theory. Based on this theory, the increase in urbanisation, as a measure of ecological change, is associated with significant cultural changes of psychology. Whereas urbanisation is linked to greater individualistic values and materialistic attitudes, rural environments are strongly associated with collectivistic values like allegiance, prevalence of religion, and feelings of belonging and benevolence. Due to an increase in the German urbanisation rate over time, our study investigates whether Germany and the German-speaking countries around show the presumed changes in psychology. By using Google Books Ngram Viewer, we find that word frequencies, signifying individualistic (collectivistic) values, are positively (negatively) related to the urbanisation rate of Germany. Our results indicate that predictions about implications of an urbanising population for the psychology of culture hold true, supporting international universality of the social change and human development theory. Furthermore, we provide evidence for a predicted reversal for the time during and after World War II, reflecting Nazi propaganda and influence.

Investigating measurement equivalence of visual analogue scales and Likert-type scales in Internet-based personality questionnaires.

Visual analogue scales (VASs) have shown superior measurement qualities in comparison to traditional Likert-type response scales in previous studies. The present study expands the comparison of response scales to properties of Internet-based personality scales in a within-subjects design. A sample of 879 participants filled out an online questionnaire measuring Conscientiousness, Excitement Seeking, and Narcissism. The questionnaire contained all instruments in both answer scale versions in a counterbalanced design. Results show comparable reliabilities, means, and SDs for the VAS versions of the original scales, in comparison to Likert-type scales. To assess the validity of the measurements, age and gender were used as criteria, because all three constructs have shown non-zero correlations with age and gender in previous research. Both response scales showed a high overlap and the proposed relationships with age and gender. The associations were largely identical, with the exception of an increase in explained variance when predicting age from the VAS version of Excitement Seeking (B10 = 1318.95, ΔR(2) = .025). VASs showed similar properties to Likert-type response scales in most cases.

The state of web-based research: A survey and call for inclusion in curricula.

The first papers that reported on conducting psychological research on the web were presented at the Society for Computers in Psychology conference 20 years ago, in 1996. Since that time, there has been an explosive increase in the number of studies that use the web for data collection. As such, it seems a good time, 20 years on, to examine the health and adoption of sound practices of research on the web. The number of studies conducted online has increased dramatically. Overall, it seems that the web can be a method for conducting valid psychological studies. However, it is less clear that students and researchers are aware of the nature of web research. While many studies are well conducted, there is also a certain laxness appearing regarding the design and conduct of online studies. This laxness appears both anecdotally to the authors as managers of large sites for posting links to online studies, and in a survey of current researchers. One of the deficiencies discovered is that there is no coherent approach to educating researchers as to the unique features of web research.

Using Visual Analogue Scales in eHealth: Non-Response Effects in a Lifestyle Intervention.

BACKGROUND: Visual analogue scales (VASs) have been shown to be valid measurement instruments and a better alternative to Likert-type scales in Internet-based research, both empirically and theoretically [1,2]. Upsides include more differentiated responses, better measurement level, and less error. Their feasibility and properties in the context of eHealth, however, have not been examined so far. OBJECTIVE: The present study examined VASs in the context of a lifestyle study conducted online, measuring the impact of VASs on distributional properties and non-response. METHOD: A sample of 446 participants with a mean age of 52.4 years (standard deviation (SD) = 12.1) took part in the study. The study was carried out as a randomized controlled trial, aimed at supporting participants over 8 weeks with an additional follow-up measurement. In addition to the randomized questionnaire, participants were further randomly assigned to either a Likert-type or VAS response scale version of the measures. RESULTS: Results showed that SDs were lower for items answered via VASs, 2P (Y ≥ 47 | n=55, P=.5) < .001. Means did not differ across versions. Participants in the VAS version showed lower dropout rates than participants in the Likert version, odds ratio = 0.75, 90% CI (0.58-0.98), P=.04. Number of missing values did not differ between questionnaire versions. CONCLUSIONS: The VAS is shown to be a valid instrument in the eHealth context, offering advantages over Likert-type scales. The results of the study provide further support for the use of VASs in Internet-based research, extending the scope to senior samples in the health context. TRIAL REGISTRATION: Clinicaltrials.gov NCT01909349; https://clinicaltrials.gov/ct2/show/NCT01909349 (Archived by WebCite at http://www.webcitation.org/6h88sLw2Y).

Simple construct evaluation with latent class analysis: An investigation of Facebook addiction and the development of a short form of the Facebook Addiction Test (F-AT).

In psychological research, there is a growing interest in using latent class analysis (LCA) for the investigation of quantitative constructs. The aim of this study is to illustrate how LCA can be applied to gain insights on a construct and to select items during test development. We show the added benefits of LCA beyond factor-analytic methods, namely being able (1) to describe groups of participants that differ in their response patterns, (2) to determine appropriate cutoff values, (3) to evaluate items, and (4) to evaluate the relative importance of correlated factors. As an example, we investigated the construct of Facebook addiction using the Facebook Addiction Test (F-AT), an adapted version of the Internet Addiction Test (I-AT). Applying LCA facilitates the development of new tests and short forms of established tests. We present a short form of the F-AT based on the LCA results and validate the LCA approach and the short F-AT with several external criteria, such as chatting, reading newsfeeds, and posting status updates. Finally, we discuss the benefits of LCA for evaluating quantitative constructs in psychological research.

Visual DMDX: A web-based authoring tool for DMDX, a Windows display program with millisecond accuracy.

DMDX is a software package for the experimental control and timing of stimulus display for Microsoft Windows systems. DMDX is reliable, flexible, millisecond accurate, and can be downloaded free of charge; therefore it has become very popular among experimental researchers. However, setting up a DMDX-based experiment is burdensome because of its command-based interface. Further, DMDX relies on RTF files in which parts of the stimuli, design, and procedure of an experiment are defined in a complicated (DMASTR-compatible) syntax. Other experiment software, such as E-Prime, Psychopy, and WEXTOR, became successful as a result of integrated visual authoring tools. Such an intuitive interface was lacking for DMDX. We therefore created and present here Visual DMDX (http://visualdmdx.com/), a HTML5-based web interface to set up experiments and export them to DMDX item files format in RTF. Visual DMDX offers most of the features available from the rich DMDX/DMASTR syntax, and it is a useful tool to support researchers who are new to DMDX. Both old and modern versions of DMDX syntax are supported. Further, with Visual DMDX, we go beyond DMDX by having added export to JSON (a versatile web format), easy backup, and a preview option for experiments. In two examples, one experiment each on lexical decision making and affective priming, we explain in a step-by-step fashion how to create experiments using Visual DMDX. We release Visual DMDX under an open-source license to foster collaboration in its continuous improvement.

Build your own social network laboratory with Social Lab: a tool for research in social media.

Social networking has surpassed e-mail and instant messaging as the dominant form of online communication (Meeker, Devitt, & Wu, 2010). Currently, all large social networks are proprietary, making it difficult to impossible for researchers to make changes to such networks for the purpose of study design and access to user-generated data from the networks. To address this issue, the authors have developed and present Social Lab, an Internet-based free and open-source social network software system available from http://www.sociallab.es . Having full availability of navigation and communication data in Social Lab allows researchers to investigate behavior in social media on an individual and group level. Automated artificial users ("bots") are available to the researcher to simulate and stimulate social networking situations. These bots respond dynamically to situations as they unfold. The bots can easily be configured with scripts and can be used to experimentally manipulate social networking situations in Social Lab. Examples for setting up, configuring, and using Social Lab as a tool for research in social media are provided.

Psychological well-being during the COVID-19 pandemic: Combining a web survey with experience sampling methodology.

COVID-19-related regulations have impacted the economy and people's well-being, highlighting the long-standing problem of inequality. This research explored how COVID-19-related restrictive policies, such as a lockdown or social distancing, affected people's well-being. In Study 1, a cross-sectional online survey (N = 685), we examined the associations between socio-economic characteristics, the number of resources, their relative change, people's stress levels, and their support of restrictive policies. We found that financial loss due to COVID-19, the number of children at home, and the intensity of restrictive measures were associated with higher stress by restrictive measures. The lower support for restrictive measures was observed among those who experienced financial loss due to COVID-19, had more children at home, less frequently accessed COVID-19-related information in the media, and did not perform self-isolation. Men were generally less supportive of restrictions than women, and the number of new COVID-19 cases was negatively related to the support. Lower stress and higher support for restrictive measures were positively associated with life satisfaction. In Study 2, an experience-sampling survey (Nparticipants = 46, Nresponses = 1112), the participants rated their well-being and level of available resources daily for one month. We observed that daily increases in well-being, characterized by higher life satisfaction and lower levels of stress and boredom, were positively associated with more social communication and being outdoors. In summary, the findings support the resource and demand framework, which states that people with access to resources can better cope with the demands of restrictive policies. Implications for policies and interventions to improve well-being are discussed.

Increasing digitalization is associated with anxiety and depression: A Google Ngram analysis.

The prevalence of anxiety disorders and depression are rising worldwide. Studies investigating risk factors on a societal level leading to these rises are so far limited to social-economic status, social capital, and unemployment, while most such studies rely on self-reports to investigate these factors. Therefore, our study aims to evaluate the impact of an additional factor on a societal level, namely digitalization, by using a linguistic big data approach. We extend related work by using the Google Books Ngram Viewer (Google Ngram) to retrieve and adjust word frequencies from a large corpus of books (8 million books or 6 percent of all books ever published) and to subsequently investigate word changes in terms of anxiety disorders, depression, and digitalization. Our analyses comprise and compare data from six languages, British English, German, Spanish, Russian, French, and Italian. We also retrieved word frequencies for the control construct "religion". Our results show an increase in word frequency for anxiety, depression, and digitalization over the last 50 years (r = .79 to .89, p < .001), a significant correlation between the frequency of anxiety and depression words (r = .98, p < .001), a significant correlation between the frequency of anxiety and digitalization words (r = .81, p < .001), and a significant correlation between the frequency of depression and anxiety words (r = .81, p < .001). For the control construct religion, we found no significant correlations for word frequency over the last 50 years and no significant correlation between the frequency of anxiety and depression words. Our results showed a negative correlation between the frequency of depression and religion words (r = -.25, p < .05). We also improved the method by excluding terms with double meanings detected by 73 independent native speakers. Implications for future research and professional and clinical implications of these findings are discussed.