Quantifying idiosyncratic and shared contributions to judgment.
Behav Res Methods
; 52(4): 1428-1444, 2020 08.
Article
in En
| MEDLINE
| ID: mdl-31898288
ABSTRACT
Identifying relative idiosyncratic and shared contributions to judgments is a fundamental challenge to the study of human behavior, yet there is no established method for estimating these contributions. Using edge cases of stimuli varying in intrarater reliability and interrater agreement-faces (high on both), objects (high on the former, low on the latter), and complex patterns (low on both)-we showed that variance component analyses (VCAs) accurately captured the psychometric properties of the data (Study 1). Simulations showed that the VCA generalizes to any arbitrary continuous rating and that both sample and stimulus set size affect estimate precision (Study 2). Generally, a minimum of 60 raters and 30 stimuli provided reasonable estimates within our simulations. Furthermore, VCA estimates stabilized given more than two repeated measures, consistent with the finding that both intrarater reliability and interrater agreement increased nonlinearly with repeated measures (Study 3). The VCA provides a rigorous examination of where variance lies in data, can be implemented using mixed models with crossed random effects, and is general enough to be useful in any judgment domain in which agreement and disagreement are important to quantify and in which multiple raters independently rate multiple stimuli.
Key words
Full text:
1
Collection:
01-internacional
Database:
MEDLINE
Main subject:
Research Design
/
Judgment
Type of study:
Prognostic_studies
Limits:
Humans
Language:
En
Journal:
Behav Res Methods
Journal subject:
CIENCIAS DO COMPORTAMENTO
Year:
2020
Document type:
Article
Affiliation country:
United States
Publication country:
EEUU
/
ESTADOS UNIDOS
/
ESTADOS UNIDOS DA AMERICA
/
EUA
/
UNITED STATES
/
UNITED STATES OF AMERICA
/
US
/
USA