A comparative evaluation of measures to assess randomness in human-generated sequences.

Whether and how well people can behave randomly is of interest in many areas of psychological research. The ability to generate randomness is often investigated using random number generation (RNG) tasks, in which participants are asked to generate a sequence of numbers that is as random as possible. However, there is no consensus on how best to quantify the randomness of responses in human-generated sequences. Traditionally, psychologists have used measures of randomness that directly assess specific features of human behavior in RNG tasks, such as the tendency to avoid repetition or to systematically generate numbers that have not been generated in the recent choice history, a behavior known as cycling. Other disciplines have proposed measures of randomness that are based on a more rigorous mathematical foundation and are less restricted to specific features of randomness, such as algorithmic complexity. More recently, variants of these measures have been proposed to assess systematic patterns in short sequences. We report the first large-scale integrative study to compare measures of specific aspects of randomness with entropy-derived measures based on information theory and measures based on algorithmic complexity. We compare the ability of the different measures to discriminate between human-generated sequences and truly random sequences based on atmospheric noise, and provide a systematic analysis of how the usefulness of randomness measures is affected by sequence length. We conclude with recommendations that can guide the selection of appropriate measures of randomness in psychological research.

More than random responding: Empirical evidence for the validity of the (Extended) Crosswise Model.

The Randomized Response Technique (Warner, Journal of the American Statistical Association, 60, 63-69, 1965) has been developed to control for socially desirable responses in surveys on sensitive attributes. The Crosswise Model (CWM; Yu et al., Metrika, 67, 251-263, 2008) and its extension, the Extended Crosswise Model (ECWM; Heck et al., Behavior Research Methods, 50, 1895-1905, 2018), are advancements of the Randomized Response Technique that have provided promising results in terms of improved validity of the obtained prevalence estimates compared to estimates based on conventional direct questions. However, recent studies have raised the question as to whether these promising results might have been primarily driven by a methodological artifact in terms of random responses rather than a successful control of socially desirable responding. The current study was designed to disentangle the influence of successful control of socially desirable responding and random answer behavior on the validity of (E)CWM estimates. To this end, we orthogonally manipulated the direction of social desirability (undesirable vs. desirable) and the prevalence (high vs. low) of sensitive attributes. Our results generally support the notion that the ECWM successfully controls social desirability bias and is inconsistent with the alternative account that ECWM estimates are distorted by a substantial influence of random responding. The results do not rule out a small proportion of random answers, especially when socially undesirable attributes with high prevalence are studied, or when high randomization probabilities are applied. Our results however do rule out that random responding is a major factor that can account for the findings attesting to the improved validity of (E)CWM as compared with DQ estimates.

Nothing but the truth? Effects of faking on the validity of the crosswise model.

In self-reports, socially desirable responding threatens the validity of prevalence estimates for sensitive personal attitudes and behaviors. Indirect questioning techniques such as the crosswise model attempt to control for the influence of social desirability bias. The crosswise model has repeatedly been found to provide more valid prevalence estimates than direct questions. We investigated whether crosswise model estimates are also less susceptible to deliberate faking than direct questions. To this end, we investigated the effect of "fake good" instructions on responses to direct and crosswise model questions. In a sample of 1,946 university students, 12-month prevalence estimates for a sensitive road traffic behavior were higher and thus presumably more valid in the crosswise model than in a direct question. Moreover, "fake good" instructions severely impaired the validity of the direct questioning estimates, whereas the crosswise model estimates were unaffected by deliberate faking. Participants also reported higher levels of perceived confidentiality and a lower perceived ease of faking in the crosswise model compared to direct questions. Our results corroborate previous studies finding the crosswise model to be an effective tool for counteracting the detrimental effects of positive self-presentation in surveys on sensitive issues.

Controlling social desirability bias: An experimental investigation of the extended crosswise model.

Indirect questioning techniques such as the crosswise model aim to control for socially desirable responding in surveys on sensitive personal attributes. Recently, the extended crosswise model has been proposed as an improvement over the original crosswise model. It offers all of the advantages of the original crosswise model while also enabling the detection of systematic response biases. We applied the extended crosswise model to a new sensitive attribute, campus islamophobia, and present the first experimental investigation including an extended crosswise model, and a direct questioning control condition, respectively. In a paper-pencil questionnaire, we surveyed 1,361 German university students using either a direct question or the extended crosswise model. We found that the extended crosswise model provided a good model fit, indicating no systematic response bias and allowing for a pooling of the data of both groups of the extended crosswise model. Moreover, the extended crosswise model yielded significantly higher estimates of campus Islamophobia than a direct question. This result could either indicate that the extended crosswise model was successful in controlling for social desirability, or that response biases such as false positives or careless responding have inflated the estimate, which cannot be decided on the basis of the available data. Our findings highlight the importance of detecting response biases in surveys implementing indirect questioning techniques.

How explicit warnings reduce the truth effect: A multinomial modeling approach.

The finding that repeating a statement typically increases its perceived truth has been referred to as the truth effect. Previous research has found that warning participants about the truth effect can successfully reduce, but not eliminate the effect. We used a multinomial modeling approach to investigate how warnings affect the cognitive processes that are assumed to underlie judgments of truth. In a laboratory experiment (N = 167), half of the participants were warned about the truth effect before judging the truth of repeated and new statements. Importantly, whereas half of the presented statements were of relatively unknown validity, participants could likely identify the correct truth status for the other half of the statements by drawing on stored knowledge. Multinomial modeling analyses revealed that warning instructions did not affect the retrieval of knowledge or participants' guessing behavior relative to a control condition. Instead, warned participants exhibited a significantly reduced tendency to rely on experiential information such as processing fluency when judging a repeated statement's truth. However, this was only the case for statements for which participants held relevant knowledge. These results are consistent with the notion that it is possible to discount metacognitive experiences such as processing ease when their informational value is questioned. Specifically, our findings suggest that people are less likely to base their judgments of truth on experiential information and metacognitive experiences induced by repetition if (a) they are warned about the deceptive power of repetition, and (b) other valid cues are available to inform their judgments.

Can detailed instructions and comprehension checks increase the validity of crosswise model estimates?

The crosswise model is an indirect questioning technique designed to control for socially desirable responding. Although the technique has delivered promising results in terms of improved validity in survey studies of sensitive issues, recent studies have indicated that the crosswise model may sometimes produce false positives. Hence, we investigated whether an insufficient understanding of the crosswise model instructions might be responsible for these false positives and whether ensuring a deeper understanding of the model and surveying more highly educated respondents reduces the problem of false positives. To this end, we experimentally manipulated the amount of information respondents received in the crosswise model instructions. We compared a crosswise model condition with only brief instructions and a crosswise model condition with detailed instructions and additional comprehension checks. Additionally, we compared the validity of crosswise model estimates between a higher- and a lower-educated subgroup of respondents. Our results indicate that false positives among highly educated respondents can be reduced when detailed instructions and comprehension checks are employed. Since false positives can also occur in direct questioning, they do not appear to be a specific flaw of the crosswise model, but rather a more general problem of self-reports on sensitive topics. False negatives were found to occur for all questioning techniques, but were less prevalent in the crosswise model than in the direct questioning condition. We highlight the importance of comprehension checks when applying indirect questioning and emphasize the necessity of developing instructions suitable for lower-educated respondents.

On the validity of non-randomized response techniques: an experimental comparison of the crosswise model and the triangular model.

Non-randomized response techniques (NRRTs) such as the crosswise model and the triangular model (CWM and TRM; Yu et al. Metrika, 67, 251-263, 2008) have been developed to control for socially desirable responding in surveys on sensitive personal attributes. We present the first study to directly compare the validity of the CWM and TRM and contrast their performance with a conventional direct questioning (DQ) approach. In a paper-pencil survey of 1382 students, we obtained prevalence estimates for two sensitive attributes (xenophobia and rejection of further refugee admissions) and one nonsensitive control attribute with a known prevalence (the first letter of respondents' surnames). Both NRRTs yielded descriptively higher prevalence estimates for the sensitive attributes than DQ; however, only the CWM estimates were significantly higher. We attribute the higher prevalence estimates for the CWM to its response symmetry, which is lacking in the TRM. Only the CWM provides symmetric answer options, meaning that there is no "safe" alternative respondents can choose to distance themselves from being carriers of the sensitive attribute. Prevalence estimates for the nonsensitive control attribute with known prevalence confirmed that neither method suffered from method-specific bias towards over- or underestimation. Exploratory moderator analyses further suggested that the sensitive attributes were perceived as more sensitive among politically left-oriented than among politically right-oriented respondents. Based on our results, we recommend using the CWM over the TRM in future studies on sensitive personal attributes.

Eyewitness identification in simultaneous and sequential lineups: an investigation of position effects using receiver operating characteristics.

For decades, sequential lineups have been considered superior to simultaneous lineups in the context of eyewitness identification. However, most of the research leading to this conclusion was based on the analysis of diagnosticity ratios that do not control for the respondent's response criterion. Recent research based on the analysis of ROC curves has found either equal discriminability for sequential and simultaneous lineups, or higher discriminability for simultaneous lineups. Some evidence for potential position effects and for criterion shifts in sequential lineups has also been reported. Using ROC curve analysis, we investigated the effects of the suspect's position on discriminability and response criteria in both simultaneous and sequential lineups. We found that sequential lineups suffered from an unwanted position effect. Respondents employed a strict criterion for the earliest lineup positions, and shifted to a more liberal criterion for later positions. No position effects and no criterion shifts were observed in simultaneous lineups. This result suggests that sequential lineups are not superior to simultaneous lineups, and may give rise to unwanted position effects that have to be considered when conducting police lineups.

PageFocus: Using paradata to detect and prevent cheating on online achievement tests.

Cheating threatens the validity of unproctored online achievement tests. To address this problem, we developed PageFocus, a JavaScript that detects when participants abandon test pages by switching to another window or browser tab. In a first study, we aimed at testing whether PageFocus could detect and prevent cheating. We asked 115 lab and 186 online participants to complete a knowledge test comprising items that were difficult to answer but easy to look up on the Internet. Half of the participants were invited to look up the solutions, which significantly increased their test scores. The PageFocus script detected test takers who abandoned the test page with very high sensitivity and specificity, and successfully reduced cheating by generating a popup message that asked participants not to cheat. In a second study, 510 online participants completed a knowledge test comprising items that could easily be looked up and a reasoning task involving matrices that were impossible to look up. In a first group, a performance-related monetary reward was promised to the top scorers; in a second group, participants took part in a lottery that provided performance-unrelated rewards; and in a third group, no incentive was offered. PageFocus revealed that participants cheated more when performance-related incentives were offered. As expected, however, this effect was limited to items that could easily be looked up. We recommend that PageFocus be routinely employed to detect and prevent cheating on online achievement tests.

On the comprehensibility and perceived privacy protection of indirect questioning techniques.

On surveys that assess sensitive personal attributes, indirect questioning aims at increasing respondents' willingness to answer truthfully by protecting confidentiality. However, the assumption that subjects understand questioning procedures fully and trust them to protect their privacy is rarely tested. In a scenario-based design, we compared four indirect questioning procedures in terms of their comprehensibility and perceived privacy protection. All indirect questioning techniques were found to be less comprehensible by respondents than a conventional direct question used for comparison. Less-educated respondents experienced more difficulties when confronted with any indirect questioning technique. Regardless of education, the crosswise model was found to be the most comprehensible among the four indirect methods. Indirect questioning in general was perceived to increase privacy protection in comparison to a direct question. Unexpectedly, comprehension and perceived privacy protection did not correlate. We recommend assessing these factors separately in future evaluations of indirect questioning.

On the importance of considering heterogeneity in witnesses' competence levels when reconstructing crimes from multiple witness testimonies.

Aggregating information across multiple testimonies may improve crime reconstructions. However, different aggregation methods are available, and research on which method is best suited for aggregating multiple observations is lacking. Furthermore, little is known about how variance in the accuracy of individual testimonies impacts the performance of competing aggregation procedures. We investigated the superiority of aggregation-based crime reconstructions involving multiple individual testimonies and whether this superiority varied as a function of the number of witnesses and the degree of heterogeneity in witnesses' ability to accurately report their observations. Moreover, we examined whether heterogeneity in competence levels differentially affected the relative accuracy of two aggregation procedures: a simple majority rule, which ignores individual differences, and the more complex general Condorcet model (Romney et al., Am Anthropol 88(2):313-338, 1986; Batchelder and Romney, Psychometrika 53(1):71-92, 1988), which takes into account differences in competence between individuals. 121 participants viewed a simulated crime and subsequently answered 128 true/false questions about the crime. We experimentally generated groups of witnesses with homogeneous or heterogeneous competences. Both the majority rule and the general Condorcet model provided more accurate reconstructions of the observed crime than individual testimonies. The superiority of aggregated crime reconstructions involving multiple individual testimonies increased with an increasing number of witnesses. Crime reconstructions were most accurate when competences were heterogeneous and aggregation was based on the general Condorcet model. We argue that a formal aggregation should be considered more often when eyewitness testimonies have to be assessed and that the general Condorcet model provides a good framework for such aggregations.

Assessing the validity of two indirect questioning techniques: A Stochastic Lie Detector versus the Crosswise Model.

Estimates of the prevalence of sensitive attributes obtained through direct questions are prone to being distorted by untruthful responding. Indirect questioning procedures such as the Randomized Response Technique (RRT) aim to control for the influence of social desirability bias. However, even on RRT surveys, some participants may disobey the instructions in an attempt to conceal their true status. In the present study, we experimentally compared the validity of two competing indirect questioning techniques that presumably offer a solution to the problem of nonadherent respondents: the Stochastic Lie Detector and the Crosswise Model. For two sensitive attributes, both techniques met the "more is better" criterion. Their application resulted in higher, and thus presumably more valid, prevalence estimates than a direct question. Only the Crosswise Model, however, adequately estimated the known prevalence of a nonsensitive control attribute.

Looking back on the London Olympics: Independent outcome and hindsight effects in decision evaluation.

Outcome bias and hindsight bias are related, but how exactly? To remedy theoretical ambiguity and non-existent directly relevant empirical research, we contrast an older idea (Baron & Hershey, 1988, J. Pers. Soc. Psychol., 54, 569) that sees outcome bias as partly mediated through hindsight bias with the idea that the two biases independently affect decision evaluations. In an Internet study of retrospections on the 2012 London Olympics, evaluations of the Games' success and its foreseeability had independent effects on evaluations of the International Olympic Committee's decision to award the Olympics to London; there was no evidence of mediation. Further theoretical discussion emphasizes the need to distinguish between a holistic assessment of decisions and a more specific assessment of the decision-making process in future outcome bias research.

cocor: a comprehensive solution for the statistical comparison of correlations.

A valid comparison of the magnitude of two correlations requires researchers to directly contrast the correlations using an appropriate statistical test. In many popular statistics packages, however, tests for the significance of the difference between correlations are missing. To close this gap, we introduce cocor, a free software package for the R programming language. The cocor package covers a broad range of tests including the comparisons of independent and dependent correlations with either overlapping or nonoverlapping variables. The package also includes an implementation of Zou's confidence interval for all of these comparisons. The platform independent cocor package enhances the R statistical computing environment and is available for scripting. Two different graphical user interfaces-a plugin for RKWard and a web interface-make cocor a convenient and user-friendly tool.

A Strong Validation of the Crosswise Model Using Experimentally-Induced Cheating Behavior.

We constructed an online cheating paradigm that could be used to validate the Crosswise Model ( Yu, Tian, & Tang, 2008 ), a promising indirect questioning technique designed to control for socially desirable responding on sensitive questions. Participants qualified for a reward only if they could identify the target words from three anagrams, one of which was virtually unsolvable as shown on a pretest. Of the 664 participants, 15.5% overreported their performance and were categorized as cheaters. When participants were asked to report whether they had cheated, a conventional direct question resulted in a substantial underestimate (5.1%) of the known prevalence of cheaters. Using a CWM question resulted in a more accurate estimate (13.0%). This result shows that the CWM can be used to control for socially desirable responding and provides estimates that are much closer to the known prevalence of a sensitive personal attribute than those obtained using a direct question.

Do sequentially-presented answer options prevent the use of testwiseness cues on continuing medical education tests?

Testwiseness--that is, the ability to find subtle cues towards the solution by the simultaneous comparison of the available answer options--threatens the validity of multiple-choice (MC) tests. Discrete-option multiple-choice (DOMC) has recently been proposed as a computerized alternative testing format for MC tests, and presumably allows for a better control of testwiseness. It is based on a sequential rather than simultaneous presentation of answer options. The test taker has to decide on the correctness of one option after another, until the item has been answered either correctly or incorrectly. Test items that have been criticized for being susceptible to testwiseness strategies are used in continuing medical education (CME) programs aimed at developing and maintaining the knowledge of medical professionals. In Experiment 1 with 48 adults without a special medical education, presenting answer options sequentially reduced the use of testwiseness cues on a CME test compared to their simultaneous presentation as shown by a significant interaction of answer format and the availability of cues (p = .01, η(2) = 0.13). This result was not dependent on a hint towards potential cues to the solution, as it also held when another 86 adults were not informed of the presence of testwiseness cues in Experiment 2 (p < .001, η(2) = 0.14). The result could be replicated with 106 medical students and medical doctors in Experiment 3. Items were answered correctly more often in the MC condition (71%) than in the DOMC condition (47%), but only when items contained cues to their solution. A significant interaction between answer format and the availability of cues (p = .02, η(2) = 0.05) showed that the sequential DOMC answer format allows for a better control of testwiseness than traditional MC testing.

The influence of relative age on the composition of professional soccer squads.

Children born shortly after the cut-off date applying for age grouping in minor sports profit from being able to remain for almost an additional year in their respective age group compared to those born later in the same year. A shift of the cut-off date in German youth soccer in 1997 allowed us to investigate the effect of a favourable relative age as well as the effect of a change of the cut-off date on the birth-date distribution of soccer players in the Bundesliga (Division 1), the highest professional soccer league in Germany. Both the cut-off date and the change of the cut-off date exerted a significant influence on the composition of professional senior squads. In the 1997/98 season, there was an overrepresentation of players profiting from the old cut-off date of 1 August. In the 2007/08 season, 10 years after the change of the cut-off date, there was an overrepresentation of players born shortly after the new cut-off date of 1 January. This pattern of results shows that owing to the cut-off dates applying in youth soccer, the date of birth of young players systematically influences their later professional success and career.

Seriousness checks are useful to improve data validity in online research.

Nonserious answering behavior increases noise and reduces experimental power; it is therefore one of the most important threats to the validity of online research. A simple way to address the problem is to ask respondents about the seriousness of their participation and to exclude self-declared nonserious participants from analysis. To validate this approach, a survey was conducted in the week prior to the German 2009 federal election to the Bundestag. Serious participants answered a number of attitudinal and behavioral questions in a more consistent and predictively valid manner than did nonserious participants. We therefore recommend routinely employing seriousness checks in online surveys to improve data validity.

Publication bias in meta-analyses of the efficacy of psychotherapeutic interventions for depression.

OBJECTIVE: The aim of this study was to assess whether systematic reviews investigating psychotherapeutic interventions for depression are affected by publication bias. Only homogeneous data sets were included, as heterogeneous data sets can distort statistical tests of publication bias. METHOD: We applied Begg and Mazumdar's adjusted rank correlation test, Egger's regression analysis, and the trim and fill procedure to assess the presence and magnitude of publication bias in all homogeneous data sets of systematic reviews published up to September 2010. RESULTS: Thirty-one data sets reported in 19 meta-analyses fulfilled our inclusion criteria. Significant bias was detected in 5 (16.13%; rank correlation test) and 6 (19.35%; Egger's regression analysis) of these data sets. Applying the trim and fill procedure to amend presumably missing studies rarely changed the assessment of the efficacy of therapeutic interventions, with 2 exceptions. In 1 data set psychotherapy was no longer found to be significantly more efficacious than pharmacotherapy in reducing dropout at posttreatment when publication bias was taken into account. In the 2nd data set, after correcting for publication bias, there was no longer evidence that depressed patients without comorbid personality disorder profited more from psychotherapy and pharmacotherapy than patients with comorbid personality disorder. CONCLUSIONS: The results suggest that taken together, psychotherapy research for depression is only marginally affected by the selective reporting of positive outcomes. With 2 notable exceptions, correcting for publication bias did not change the evaluation of the efficacy of psychotherapeutic interventions.

Estimating the prevalence of nonpaternity in Germany.

The prevalence of nonpaternity in human societies is difficult to establish. To obtain a current and fairly unbiased estimate of the nonpaternity rate in Germany, we analysed a dataset consisting of 971 children and their parents in whom human leukocyte antigen (HLA) typing had been carried out in the context of bone marrow transplantation. In this sample, nine exclusions (0.93%) could be identified on the basis of more than 300 HLA-haplotypes defined by four HLA genes. Given this number of exclusions, a maximum likelihood estimate of the nonpaternity rate in the population of 0.94% was obtained with asymptotic 95% confidence limits of 0.33% and 1.55%, respectively. This result is in accordance with recent surveys as well as findings from Switzerland for a comparable sample, and it suggests that earlier estimates of the nonpaternity rate which were often in excess of 10% may have been largely exaggerated.