Central tendency and matched difference approaches for assessing interrater agreement.

In Study 1 of this two-part investigation, we present a "central tendency approach" and procedures for assessing overall interrater agreement across multiple groups. We define parameters for mean group agreement and construct bootstrapped confidence intervals around the mean population parameters for rWG, AD, and ICC(1). In Study 2, we extend assessments of overall interrater agreement by developing a "matched difference approach" and procedures for assessing real versus pseudo agreement in a sample of groups. Here, we use random group resampling and the matched difference between assessments of the respective rWG, AD, and ICC(1) values for actual and pseudo groups, with the establishment of bootstrapped confidence intervals around such differences. In both studies, we employ simulated and real data to demonstrate the accuracy and practical utility of the new procedures for assessing agreement with respect to groups. Notably, to generate simulated data for Studies 1 and 2, we developed a new underlying model for multilevel data and procedure for data generation, and we discuss its potential utility for enhancing research in group-level studies. Moreover, we discuss, relative to current practices, how and why the new inference procedures provide information about mean interrater agreement in the population, which can improve data aggregation decisions and interpretations of findings from group-level studies. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Statistical significance criteria for the rWG and average deviation interrater agreement indices.

Despite the widespread use of interrater agreement statistics for multilevel modeling and other types of research, the existing guidelines for inferring the statistical significance of interrater agreement are quite limited. They are largely relevant only under conditions that numerous researchers have argued rarely exist. Here we address this problem by generating guidelines for inferring statistical significance under a number of conditions via a computer simulation. As a set, these guidelines cover many of the conditions researchers commonly face. We discuss how researchers can use the guidelines presented to more reasonably infer the statistical significance of interrater agreement relative to using the limited guidelines available in the extant literature.

The dread factor: how hazards and safety training influence learning and performance.

On the basis of hypotheses derived from social and experiential learning theories, we meta-analytically investigated how safety training and workplace hazards impact the development of safety knowledge and safety performance. The results were consistent with an expected interaction between the level of engagement of safety training and hazardous event/exposure severity in the promotion of safety knowledge and performance. For safety knowledge and safety performance, highly engaging training was considerably more effective than less engaging training when hazardous event/exposure severity was high, whereas highly and less engaging training had comparable levels of effectiveness when hazardous event/exposure severity was low. Implications of these findings for theory testing and incorporating information on objective risk into workplace safety research and practice are discussed.

When birds of a feather flock together and when they do not: status composition, social dominance orientation, and organizational attractiveness.

Although similarity-attraction notions suggest that similarity--for example, in terms of values, personality, and demography--attracts, the authors found that sometimes demographic similarity attracts and sometimes it repels. Consistent with social dominance theory (J. Sidanius & F. Pratto, 1999), they demonstrated in 3 studies that when prospective employees supported group-based social hierarchies (i.e., were high in social dominance orientation), those in high-status groups were attracted to demographic similarity within an organization, whereas those in low-status groups were repelled by it. An important theoretical implication of the findings is that social dominance theory and traditional similarity-attraction notions together help explain a more complex relationship between demographic similarity and attraction than was previously acknowledged in the organizational literature.

Relative effectiveness of worker safety and health training methods.

OBJECTIVES: We sought to determine the relative effectiveness of different methods of worker safety and health training aimed at improving safety knowledge and performance and reducing negative outcomes (accidents, illnesses, and injuries). METHODS: Ninety-five quasi-experimental studies (n=20991) were included in the analysis. Three types of intervention methods were distinguished on the basis of learners' participation in the training process: least engaging (lecture, pamphlets, videos), moderately engaging (programmed instruction, feedback interventions), and most engaging (training in behavioral modeling, hands-on training). RESULTS: As training methods became more engaging (i.e., requiring trainees' active participation), workers demonstrated greater knowledge acquisition, and reductions were seen in accidents, illnesses, and injuries. All methods of training produced meaningful behavioral performance improvements. CONCLUSIONS: Training involving behavioral modeling, a substantial amount of practice, and dialogue is generally more effective than other methods of safety and health training. The present findings challenge the current emphasis on more passive computer-based and distance training methods within the public health workforce.

Accurate tests of statistical significance for r(WG) and average deviation interrater agreement indexes.

The authors demonstrated that the most common statistical significance test used with r(WG)-type interrater agreement indexes in applied psychology, based on the chi-square distribution, is flawed and inaccurate. The chi-square test is shown to be extremely conservative even for modest, standard significance levels (e.g., .05). The authors present an alternative statistical significance test, based on Monte Carlo procedures, that produces the equivalent of an approximate randomization test for the null hypothesis that the actual distribution of responding is rectangular and demonstrate its superiority to the chi-square test. Finally, the authors provide tables of critical values and offer downloadable software to implement the approximate randomization test for r(WG)-type and for average deviation (AD)-type interrater agreement indexes. The implications of these results for studying a broad range of interrater agreement problems in applied psychology are discussed.