Statistical inference for agreement between multiple raters on a binary scale.
Br J Math Stat Psychol
; 77(2): 245-260, 2024 May.
Article
in En
| MEDLINE
| ID: mdl-38233946
ABSTRACT
Agreement studies often involve more than two raters or repeated measurements. In the presence of two raters, the proportion of agreement and of positive agreement are simple and popular agreement measures for binary scales. These measures were generalized to agreement studies involving more than two raters with statistical inference procedures proposed on an empirical basis. We present two alternatives. The first is a Wald confidence interval using standard errors obtained by the delta method. The second involves Bayesian statistical inference not requiring any specific Bayesian software. These new procedures show better statistical behaviour than the confidence intervals initially proposed. In addition, we provide analytical formulas to determine the minimum number of persons needed for a given number of raters when planning an agreement study. All methods are implemented in the R package simpleagree and the Shiny app simpleagree.
Key words
Full text:
1
Collection:
01-internacional
Database:
MEDLINE
Main subject:
Software
/
Models, Statistical
Language:
En
Journal:
Br J Math Stat Psychol
Year:
2024
Document type:
Article
Affiliation country:
Países Bajos