ABSTRACT
The objective of this paper was to estimate the inter-rater reliability of expert assessments of occupational exposures. An inter-rater reliability sub-study was conducted within a population-based case-control study of postmenopausal breast cancer. Detailed information on lifetime occupational histories was obtained from participants and two industrial hygienists assigned exposures to 185 jobs using a checklist of 293 agents. Experts rated exposure for each job-agent combination according to exposure status (unexposed/exposed), confidence that the exposure occurred (possible/probable/definite), intensity (low/medium/high), and frequency (% time per week). The statistical unit of observation was each job-agent assessment (185 jobs × 293 agents = 54,205 assessments per expert). Crude agreement, Gwet AC1/2 statistics, and Cohen's Kappa were used to estimate inter-rater agreement for confidence and intensity; for frequency, the intra-class correlation coefficient (ICC) was used. The majority of job-agent combinations were evaluated by the two experts to be not exposed (crude agreement >98% of decisions). The degree of agreement between the experts for the confidence of exposure status was Gwet AC1/2 = 0.99 (95% CI: 0.99-0.99), and for intensity, a Gwet AC2 = 0.99 (95% CI: 0.99-0.99). For frequency, an ICC of 0.31 (95% CI: 0.26-0.35) was found. A sub-analysis restricted to job-agent combinations for which the two experts agreed on exposure status revealed a moderate agreement for confidence of exposure (Gwet AC2 = 0.66) and high agreement for intensity (Gwet AC2 = 0.96). For frequency, the ICC was 0.52 (95% CI: 0.47-0.57). A high level of inter-rater agreement was found for identifying exposures and for coding intensity, but agreement was lower for the coding of frequency of exposure.