Meta-analysis in a digitalized world: A step-by-step primer.

In recent years, much research and many data sources have become digital. Some advantages of digital or Internet-based research, compared to traditional lab research (e.g., comprehensive data collection and storage, availability of data) are ideal for an improved meta-analyses approach.In the meantime, in meta-analyses research, different types of meta-analyses have been developed to provide research syntheses with accurate quantitative estimations. Due to its rich and unique palette of corrections, we recommend to using the Schmidt and Hunter approach for meta-analyses in a digitalized world. Our primer shows in a step-by-step fashion how to conduct a high quality meta-analysis considering digital data and highlights the most obvious pitfalls (e.g., using only a bare-bones meta-analysis, no data comparison) not only in aggregation of the data, but also in the literature search and coding procedure which are essential steps in any meta-analysis. Thus, this primer of meta-analyses is especially suited for a situation where much of future research is headed to: digital research. To map Internet-based research and to reveal any research gap, we further synthesize meta-analyses on Internet-based research (15 articles containing 24 different meta-analyses, on 745 studies, with 1,601 effect sizes), resulting in the first mega meta-analysis of the field. We found a lack of individual participant data (e.g., age and nationality). Hence, we provide a primer for high-quality meta-analyses and mega meta-analyses that applies to much of coming research and also basic hands-on knowledge to conduct or judge the quality of a meta-analyses in a digitalized world.

Illustrating psychometric tests, scales, and constructs: An R package for Item Pool Visualization.

Researchers assessing psychological constructs have to understand and choose between several competing measures. Item Pool Visualization (IPV, Dantlgraber et al., 2019) was developed to offer a systematic and detailed portrayal of the actual content and internal balance of competing measures. To enable the use of IPV, we developed and present here the IPV R package. Its aim is to allow researchers to add IPV to their repertoire with minimal effort. Creating IPV charts from raw data requires two simple function calls, because the package streamlines model specification, model estimation, and chart creation. It improves IPV conceptually by introducing the aggregate center distance and the item overview chart. It provides many customization options and generates high-quality, vector-based PDF output. The workflow of the package is explained using a reproducible open data example from a personality assessment.

Geofencing in location-based behavioral research: Methodology, challenges, and implementation.

This manuscript presents a novel geofencing method in behavioral research. Geofencing, built upon geolocation technology, constitutes virtual fences around specific locations. Every time a participant crosses the virtual border around the geofenced area, an event can be triggered on a smartphone, e.g., the participant may be asked to complete a survey. The geofencing method can alleviate the problems of constant location tracking, such as recording sensitive geolocation information and battery drain. In scenarios where locations for geofencing are determined by participants (e.g., home, workplace), no location data need to be transferred to the researcher, so this method can ensure privacy and anonymity. Given the widespread use of smartphones and mobile Internet, geofencing has become a feasible tool in studying human behavior and cognition outside of the laboratory. The method can help advance theoretical and applied psychological science at a new frontier of context-aware research. At the same time, there is a lack of guidance on how and when geofencing can be applied in research. This manuscript aims to fill the gap and ease the adoption of the geofencing method. We describe the current challenges and implementations in geofencing and present three empirical studies in which we evaluated the geofencing method using the Samply application, a tool for mobile experience sampling research. The studies show that sensitivity and precision of geofencing were affected by the type of event, location radius, environment, operating system, and user behavior. Potential implications and recommendations for behavioral research are discussed.

Barriers and opportunities for implementation of a brief psychological intervention for post-ICU mental distress in the primary care setting - results from a qualitative sub-study of the PICTURE trial.

BACKGROUND: The results of critical illness and life-saving invasive measures during intensive care unit treatment can sometimes lead to lasting physical and psychological impairments. A multicentre randomized controlled trial from Germany (PICTURE) aims to test a brief psychological intervention, based on narrative exposure therapy, for post-traumatic stress disorder symptoms following intensive care unit treatment in the primary care setting. A qualitative analysis was conducted to understand feasibility and acceptance of the intervention beyond quantitative analysis of the main outcomes in the primary study. METHODS: Qualitative explorative sub-study of the main PICTURE trial, with eight patients from the intervention group recruited for semi-structured telephone interviews. Transcriptions were analysed according to Mayring's qualitative content analysis. Contents were coded and classified into emerging categories. RESULTS: The study population was 50% female and male, with a mean age of 60.9 years and transplantation surgery being the most frequent admission diagnosis. Four main factors were identified as conducive towards implementation of a short psychological intervention in a primary care setting: 1) long-term trustful relationship between patient and GP team; 2) intervention applied by a medical doctor; 3) professional emotional distance of the GP team; 4) brevity of the intervention. CONCLUSION: The primary setting has certain qualities such as a long-term doctor-patient relationship and low-threshold consultations that offer good opportunities for implementation of a brief psychological intervention for post-intensive care unit impairments. Structured follow-up guidelines for primary care following intensive care unit treatment are needed. Brief general practice-based interventions could be part of a stepped-care approach. TRIAL REGISTRATION: The main trial was registered at the DRKS (German Register of Clinical Trials: DRKS00012589) on 17/10/2017.

Increasing digitalization is associated with anxiety and depression: A Google Ngram analysis.

The prevalence of anxiety disorders and depression are rising worldwide. Studies investigating risk factors on a societal level leading to these rises are so far limited to social-economic status, social capital, and unemployment, while most such studies rely on self-reports to investigate these factors. Therefore, our study aims to evaluate the impact of an additional factor on a societal level, namely digitalization, by using a linguistic big data approach. We extend related work by using the Google Books Ngram Viewer (Google Ngram) to retrieve and adjust word frequencies from a large corpus of books (8 million books or 6 percent of all books ever published) and to subsequently investigate word changes in terms of anxiety disorders, depression, and digitalization. Our analyses comprise and compare data from six languages, British English, German, Spanish, Russian, French, and Italian. We also retrieved word frequencies for the control construct "religion". Our results show an increase in word frequency for anxiety, depression, and digitalization over the last 50 years (r = .79 to .89, p < .001), a significant correlation between the frequency of anxiety and depression words (r = .98, p < .001), a significant correlation between the frequency of anxiety and digitalization words (r = .81, p < .001), and a significant correlation between the frequency of depression and anxiety words (r = .81, p < .001). For the control construct religion, we found no significant correlations for word frequency over the last 50 years and no significant correlation between the frequency of anxiety and depression words. Our results showed a negative correlation between the frequency of depression and religion words (r = -.25, p < .05). We also improved the method by excluding terms with double meanings detected by 73 independent native speakers. Implications for future research and professional and clinical implications of these findings are discussed.

Psychological well-being during the COVID-19 pandemic: Combining a web survey with experience sampling methodology.

COVID-19-related regulations have impacted the economy and people's well-being, highlighting the long-standing problem of inequality. This research explored how COVID-19-related restrictive policies, such as a lockdown or social distancing, affected people's well-being. In Study 1, a cross-sectional online survey (N = 685), we examined the associations between socio-economic characteristics, the number of resources, their relative change, people's stress levels, and their support of restrictive policies. We found that financial loss due to COVID-19, the number of children at home, and the intensity of restrictive measures were associated with higher stress by restrictive measures. The lower support for restrictive measures was observed among those who experienced financial loss due to COVID-19, had more children at home, less frequently accessed COVID-19-related information in the media, and did not perform self-isolation. Men were generally less supportive of restrictions than women, and the number of new COVID-19 cases was negatively related to the support. Lower stress and higher support for restrictive measures were positively associated with life satisfaction. In Study 2, an experience-sampling survey (Nparticipants = 46, Nresponses = 1112), the participants rated their well-being and level of available resources daily for one month. We observed that daily increases in well-being, characterized by higher life satisfaction and lower levels of stress and boredom, were positively associated with more social communication and being outdoors. In summary, the findings support the resource and demand framework, which states that people with access to resources can better cope with the demands of restrictive policies. Implications for policies and interventions to improve well-being are discussed.

A Pilot Study of Application of the Stroke Riskometer Mobile App for Assessment of the Course and Clinical Outcomes of COVID-19 among Hospitalized Patients.

INTRODUCTION: Early determination of COVID-19 severity and health outcomes could facilitate better treatment of patients. Different methods and tools have been developed for predicting outcomes of COVID-19, but they are difficult to use in routine clinical practice. METHODS: We conducted a prospective cohort study of inpatients aged 20-92 years, diagnosed with COVID-19 to determine whether their individual 5-year absolute risk of stroke at the time of hospital admission predicts the course of COVID-19 severity and mortality. The risk of stroke was determined by the Stroke Riskometer mobile application. RESULTS: We examined 385 patients hospitalized with COVID-19 (median age 61 years). The participants were categorized based on COVID-19 severity: 271 (70.4%) to the "not severe" and 114 (29.6%) to the "severe" groups. The median risk of stroke the next day after hospitalization was significantly higher among patients in the severe group (2.83, 95% CI: 2.35-4.68) versus the not severe group (1.11, 95% CI: 1.00-1.29). The median risk of stroke and median systolic blood pressure (SBP) were significantly higher among non-survivors (12.04, 95% CI: 2.73-21.19) and (150, 95% CI: 140-170) versus survivors (1.31, 95% CI: 1.14-1.52) and (134, 95% CI: 130-135), respectively. Those who spent more than 2.5 h a week on physical activity were 3.1 times more likely to survive from COVID-19. Those who consumed more than one standard alcohol drink a day, or suffered with atrial fibrillation, or had poor memory were 2.5, 2.3, and 2.6 times more likely not to survive from COVID-19, respectively. CONCLUSIONS: High risk of stroke, physical inactivity, alcohol intake, high SBP, and atrial fibrillation are associated with severity and mortality of COVID-19. Our findings suggest that the Stroke Riskometer app could be used as a simple predictive tool of COVID-19 severity and mortality.

[Psychometric Properties of a German Translation of the Parental Bonding Instrument]. / Psychometrische Kennwerte einer deutschen Übersetzung des Parental Bonding Instrument.

Parenting behavior affects a child's development as well as the etiology and treatment of mental disorders. The Parental Bonding Instrument (PBI; Parker, Tupling & Brown, 1979) is a well-known measurement tool to retrospectively assess parenting styles. Yet, no sufficiently validated German version exists to date. Therefore, we developed an updated translation of the German PBI version (PBI-dt) and analyzed its psychometric properties in an online survey based on a sample of n=791 German-speaking participants with a focus on item and reliability characteristics, construct and criterion validity as well as factorial structure of the PBI-dt.Our results indicated good item characteristics and reliability (α=0.86-0.95). Correlations between PBI and CTQ-SF (Childhood Trauma Questionnaire Short Form) scales were in line with the literature. Significant differences in the reported parenting style were found between people with and without mental illness as well as between normal-weight and overweight people. These results indicated the presence of good construct and criterion validity. Confirmatory factor analyses indicated an acceptable model fit for all fit indices in the original 2-factor model of Parker et al. (1979) as well as in the 3-factor model with the scales CareCareCareCareCareCare, Discouragement of behavioral freedomDiscouragement of behavioral freedomDiscouragement of behavioral freedomDiscouragement of behavioral freedomDiscouragement of behavioral freedomDiscouragement of behavioral freedom and Denial of psychological autonomyDenial of psychological autonomyDenial of psychological autonomyDenial of psychological autonomyDenial of psychological autonomyDenial of psychological autonomy. A 3-factor structure provided additional information, e. g., a better differentiation between normal and overweight people. Hence, this German translation of the PBI has good psychometric properties and is a reliable measuring instrument.

Samply: A user-friendly smartphone app and web-based means of scheduling and sending mobile notifications for experience-sampling research.

Undertaking an experience-sampling study via smartphones is complex. Scheduling and sending mobile notifications often requires the use of proprietary software that imposes limits on participants' operating systems (whether iOS or Android) or the types of questions that can be asked via the application. We have developed an open-source platform-Samply-which overcomes these limitations. Researchers can access the entire interface via a browser, manage studies, schedule and send notifications linking to online surveys or experiments created in any Internet-based service or software, and monitor participants' responses-all without the coding skills usually needed to program a native mobile application. Participants can download the Samply Research mobile application for free from Google Play or the App Store, join a specific study, receive notifications and web links to surveys or experiments, and track their involvement. The mobile application leverages the power of the React Native JavaScript library, which allows it to be rendered in the native code of Android and iOS mobile operating systems. We describe Samply, provide a step-by-step example of conducting an experience-sampling study, and present the results of two validation studies. Study 1 demonstrates how we improved the website's usability for researchers. Study 2 validates the mobile application's data recording ability by analyzing a survey's participation rate. The application's possible limitations and how mobile device settings might affect its reliability are discussed.

Smartphone sensor accuracy varies from device to device in mobile research: The case of spatial orientation.

Smartphone usage is increasing around the globe-in daily life and as a research device in behavioral science. Smartphones offer the possibility to gather longitudinal data at little cost to researchers and participants. They provide the option to verify self-report data with data from sensors built into most smartphones. How accurate this sensor data is when gathered via different smartphone devices, e.g., in a typical experience sampling framework, has not been investigated systematically. With the present study, we investigated the accuracy of orientation data about the spatial position of smartphones via a newly invented measurement device, the RollPitcher. Objective status of pitch (vertical orientation) and roll (horizontal orientation) of the smartphone was compared to data gathered from the sensors via web browsers and native apps. Bayesian ANOVAs confirmed that the deviations in pitch and roll differed between smartphone models, with mean inaccuracies per device of up to 2.1° and 6.6°, respectively. The inaccuracies for measurements of roll were higher than for pitch, d = .28, p < .001. Our results confirm the presence of heterogeneities when gathering orientation data from different smartphone devices. In most cases, measurement via a web browser was identical to measurement via a native app, but this was not true for all smartphone devices. As a solution to lack of sensor accuracy, we recommend the development and implementation of a coherent research framework and also discuss the implications of the heterogeneities in orientation data for different research designs.

Taking the Test Taker's Perspective: Response Process and Test Motivation in Multidimensional Forced-Choice Versus Rating Scale Instruments.

The multidimensional forced-choice (MFC) format has been proposed as an alternative to the rating scale (RS) response format. However, it is unclear how changing the response format may affect the response process and test motivation of participants. In Study 1, we investigated the MFC response process using the think-aloud technique. In Study 2, we compared test motivation between the RS format and different versions of the MFC format (presenting 2, 3, 4, and 5 items simultaneously). The response process to MFC item blocks was similar to the RS response process but involved an additional step of weighing the items within a block against each other. The RS and MFC response format groups did not differ in their test motivation. Thus, from the test taker's perspective, the MFC format is somewhat more demanding to respond to, but this does not appear to decrease test motivation.

Conceptual fluency in inductive reasoning.

Psychological effects connected with fluent processing are called fluency effects. In a sample of 403 participants we test whether conceptual fluency effects can be found in the context of inductive reasoning, a context that has not been investigated before. As a conceptual manipulation we vary the use of symbols (persons and crosses) in reasoning tasks. These symbols were chosen to provide hints for the solution of the implemented tasks and thus manipulate fluency. We found evidence that these hints influence ease of processing. The proportion of solved tasks increased by 11% on average in the condition with conceptual hints, F(1,399) = 13.47, partial η2 = .033, p < .001. However, we did not find an effect of the conceptual manipulation on the temporal perception of the task. In a second study (n = 62) we strengthened our findings by investigating solution strategies for the tasks in more detail, 79% of the participants described the tasks in a way they were intended. Our results illustrate the advantages of the separation of ease of processing, fluency experience, and judgments.

Guideline for improving the reliability of Google Ngram studies: Evidence from religious terms.

The Google Books Ngram Viewer (Google Ngram) is a search engine that charts word frequencies from a large corpus of books and thereby allows for the examination of cultural change as it is reflected in books. While the tool's massive corpus of data (about 8 million books or 6% of all books ever published) has been used in various scientific studies, concerns about the accuracy of results have simultaneously emerged. This paper reviews the literature and serves as a guideline for improving Google Ngram studies by suggesting five methodological procedures suited to increase the reliability of results. In particular, we recommend the use of (I) different language corpora, (II) cross-checks on different corpora from the same language, (III) word inflections, (IV) synonyms, and (V) a standardization procedure that accounts for both the influx of data and unequal weights of word frequencies. Further, we outline how to combine these procedures and address the risk of potential biases arising from censorship and propaganda. As an example of the proposed procedures, we examine the cross-cultural expression of religion via religious terms for the years 1900 to 2000. Special emphasis is placed on the situation during World War II. In line with the strand of literature that emphasizes the decline of collectivistic values, our results suggest an overall decrease of religion's importance. However, religion re-gains importance during times of crisis such as World War II. By comparing the results obtained through the different methods, we illustrate that applying and particularly combining our suggested procedures increase the reliability of results and prevents authors from deriving wrong assumptions.

The SNARC and MARC effects measured online: Large-scale assessment methods in flexible cognitive effects.

The Spatial-Numerical Association of Response Codes (SNARC) effect (i.e., faster reactions to small/large numbers on the left-/right-hand side) is usually observed along with the linguistic Markedness of Response Codes (MARC) effect-that is, faster left-/right-hand responses to odd/even numbers. The SNARC effect is one of the most thoroughly investigated phenomena in numerical cognition. However, almost all SNARC and MARC studies to date were conducted with sample sizes smaller than 100. Here we report on a study with 1,156 participants from various linguistic and cultural backgrounds performing a typical parity judgment task. We investigated whether (1) the SNARC and MARC effects can be observed in an online setup, (2) the properties of these effects observed online are similar to those observed in laboratory setups, (3) the effects are reliable, and (4) they are valid. We found robust SNARC and MARC effects. Their magnitude and reliabilities were comparable to values previously reported in in-lab studies. Furthermore, we reproduced commonly observed validity correlations of the SNARC and MARC effects. Namely, SNARC and MARC correlated with mean reaction times and intraindividual variability in reaction times. Additionally, we found interindividual differences in the SNARC and MARC effects (e.g., finger-counting routines for the SNARC and handedness for the MARC). Large-scale testing via web-based data acquisition not only produces SNARC and MARC effects and validity correlations similar to those from small, in-lab studies, but also reveals substantial insights with regard to interindividual differences that usually cannot be revealed in the offline laboratory, due to power considerations.

Social Desirability in Spouse Ratings.

Whether or not socially desirable responding is a cause for concern in personality assessment has long been debated. For many researchers, McCrae and Costa laid the issue to rest when they showed that correcting for socially desirable responding in self-reports did not improve the agreement with spouse ratings on the Neuroticism, Extraversion, and Openness to Experience Personality Inventory. However, their findings rest on the assumption that observer ratings in general, and spouse ratings in particular, are an unbiased external criterion. If spouse ratings are also susceptible to socially desirable responding, correcting for the bias in self-rated measures cannot be assumed to increase agreement between self-reports and spouse ratings, and thus failure to do so should not be taken as evidence for the ineffectiveness of measuring and correcting for socially desirable responding. In the present study, McCrae and Costa's influential study was replicated with the exception of measuring socially desirable responding with the Marlowe-Crowne Social Desirability Scale, in both self-reports and spouse ratings. Analyses were based on responses from 70 couples who had lived together for at least one year. The results showed that both self-reports and spouse ratings are susceptible to socially desirable responding and thus McCrae and Costa's conclusion is drawn into question.

Individual differences influence two-digit number processing, but not their analog magnitude processing: a large-scale online study.

Symbolic magnitude comparison is one of the most well-studied cognitive processes in research on numerical cognition. However, while the cognitive mechanisms of symbolic magnitude processing have been intensively studied, previous studies have paid less attention to individual differences influencing symbolic magnitude comparison. Employing a two-digit number comparison task in an online setting, we replicated previous effects, including the distance effect, the unit-decade compatibility effect, and the effect of cognitive control on the adaptation to filler items, in a large-scale study in 452 adults. Additionally, we observed that the most influential individual differences were participants' first language, time spent playing computer games and gender, followed by reported alcohol consumption, age and mathematical ability. Participants who used a first language with a left-to-right reading/writing direction were faster than those who read and wrote in the right-to-left direction. Reported playing time for computer games was correlated with faster reaction times. Female participants showed slower reaction times and a larger unit-decade compatibility effect than male participants. Participants who reported never consuming alcohol showed overall slower response times than others. Older participants were slower, but more accurate. Finally, higher grades in mathematics were associated with faster reaction times. We conclude that typical experiments on numerical cognition that employ a keyboard as an input device can also be run in an online setting. Moreover, while individual differences have no influence on domain-specific magnitude processing-apart from age, which increases the decade distance effect-they generally influence performance on a two-digit number comparison task.

Best practices: Two Web-browser-based methods for stimulus presentation in behavioral experiments with high-resolution timing requirements.

The Web is a prominent platform for behavioral experiments, for many reasons (relative simplicity, ubiquity, and accessibility, among others). Over the last few years, many behavioral and social scientists have conducted Internet-based experiments using standard web technologies, both in native JavaScript and using research-oriented frameworks. At the same time, vendors of widely used web browsers have been working hard to improve the performance of their software. However, the goals of browser vendors do not always coincide with behavioral researchers' needs. Whereas vendors want high-performance browsers to respond almost instantly and to trade off accuracy for speed, researchers have the opposite trade-off goal, wanting their browser-based experiments to exactly match the experimental design and procedure. In this article, we review and test some of the best practices suggested by web-browser vendors, based on the features provided by new web standards, in order to optimize animations for browser-based behavioral experiments with high-resolution timing requirements. Using specialized hardware, we conducted four studies to determine the accuracy and precision of two different methods. The results using CSS animations in web browsers (Method 1) with GPU acceleration turned off showed biases that depend on the combination of browser and operating system. The results of tests on the latest versions of GPU-accelerated web browsers showed no frame loss in CSS animations. The same happened in many, but not all, of the tests conducted using requestAnimationFrame (Method 2) instead of CSS animations. Unbeknownst to many researchers, vendors of web browsers implement complex technologies that result in reduced quality of timing. Therefore, behavioral researchers interested in timing-dependent procedures should be cautious when developing browser-based experiments and should test the accuracy and precision of the whole experimental setup (web application, web browser, operating system, and hardware).

Questions on honest responding.

This article presents a new method for reducing socially desirable responding in Internet self-reports of desirable and undesirable behavior. The method is based on moving the request for honest responding, often included in the introduction to surveys, to the questioning phase of the survey. Over a quarter of Internet survey participants do not read survey instructions, and therefore, instead of asking respondents to answer honestly, they were asked whether they responded honestly. Posing the honesty message in the form of questions on honest responding draws attention to the message, increases the processing of it, and puts subsequent questions in context with the questions on honest responding. In three studies (nStudy I = 475, nStudy II = 1,015, nStudy III = 899), we tested whether presenting the questions on honest responding before questions on desirable and undesirable behavior could increase the honesty of responses, under the assumption that less attribution of desirable behavior and/or admitting to more undesirable behavior could be taken to indicate more honest responses. In all studies the participants who were presented with the questions on honest responding before questions on the target behavior produced, on average, significantly less socially desirable responses, though the effect sizes were small in all cases (Cohen's d ranging between 0.02 and 0.28 for single items, and from 0.17 to 0.34 for sum scores). The overall findings and the possible mechanisms behind the influence of the questions concerning honest responding on subsequent questions are discussed, and suggestions are made for future research.

The changing psychology of culture in German-speaking countries: A Google Ngram study.

This article provides evidence for the long-term affiliation between ecological and cultural changes in German-speaking countries, based on the assumptions derived from social change and human development theory. Based on this theory, the increase in urbanisation, as a measure of ecological change, is associated with significant cultural changes of psychology. Whereas urbanisation is linked to greater individualistic values and materialistic attitudes, rural environments are strongly associated with collectivistic values like allegiance, prevalence of religion, and feelings of belonging and benevolence. Due to an increase in the German urbanisation rate over time, our study investigates whether Germany and the German-speaking countries around show the presumed changes in psychology. By using Google Books Ngram Viewer, we find that word frequencies, signifying individualistic (collectivistic) values, are positively (negatively) related to the urbanisation rate of Germany. Our results indicate that predictions about implications of an urbanising population for the psychology of culture hold true, supporting international universality of the social change and human development theory. Furthermore, we provide evidence for a predicted reversal for the time during and after World War II, reflecting Nazi propaganda and influence.

Can smartphones be used to bring computer-based tasks from the lab to the field? A mobile experience-sampling method study about the pace of life.

Researchers are increasingly using smartphones to collect scientific data. To date, most smartphone studies have collected questionnaire data or data from the built-in sensors. So far, few studies have analyzed whether smartphones can also be used to conduct computer-based tasks (CBTs). Using a mobile experience-sampling method study and a computer-based tapping task as examples (N = 246; twice a day for three weeks, 6,000+ measurements), we analyzed how well smartphones can be used to conduct a CBT. We assessed methodological aspects such as potential technologically induced problems, dropout, task noncompliance, and the accuracy of millisecond measurements. Overall, we found few problems: Dropout rate was low, and the time measurements were very accurate. Nevertheless, particularly at the beginning of the study, some participants did not comply with the task instructions, probably because they did not read the instructions before beginning the task. To summarize, the results suggest that smartphones can be used to transfer CBTs from the lab to the field, and that real-world variations across device manufacturers, OS types, and CPU load conditions did not substantially distort the results.